You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "peter-toth (via GitHub)" <gi...@apache.org> on 2026/09/17 16:09:03 UTC
[PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown via splicing [spark]
peter-toth opened a new pull request, #58895:
URL: https://github.com/apache/spark/pull/58895
### What changes were proposed in this pull request?
This adds a late-materialization read path to the vectorized Parquet reader, so the optimizer can push a runtime filter into the scan and have it prune value-column IO instead of running as a post-scan `FilterExec`. The filter it pushes today is `BloomFilterMightContain`, the runtime bloom `InjectRuntimeFilter` builds for a join.
The reader reads the filter's key columns first, evaluates the predicate per row, and then reads the remaining columns restricted to the surviving row ranges. Each output batch is spliced together from the key vectors it kept and the value vectors it read for those rows.
**Planning.**
- New SQL conf `spark.sql.parquet.storageFilterPushdown.enabled`, default `false`, session-bound. It is a planning-time decision only. With it off, no storage filter is attached to a scan in the first place and the bloom stays where it is today.
- `FileSourceStrategy.extractStorageFilters` lifts eligible top-level `BloomFilterMightContain` conjuncts out of `afterScanFilters` into a new `storageFilters` slot on `FileSourceScanExec`. Eligibility is checked in full at planning time, because extraction removes the conjunct from the post-scan `Filter` and nothing else would apply it afterwards. A conjunct qualifies when the file format is exactly `ParquetFileFormat`, the vectorized reader is feasible for `partitionSchema ++ outputDataSchema`, the conjunct is deterministic, and every reference on the bloom's value side is a projected data column of a type the reader can copy.
- `ParquetStorageFilter.isSupportedKeyType` is the single authority on key-column types. Both `extractStorageFilters` and `ParquetStorageFilter.create` consult it, and it lists exactly the types `VectorizedParquetRecordReader.copierFor` handles. It is deliberately narrower than `AtomicType`, since `VariantType`, `GeometryType` and `GeographyType` are atomic but have no primitive Parquet leaf for phase 1 to read into.
- `FileSourceScanLike` gains `storageFilters: Seq[Expression]` and five SQL metrics, described below. The `StorageFilters` entry in the scan description is only emitted when the scan has storage filters, so explain output is unchanged for everyone else.
- `FileSourceScanExec.preparedStorageFilters` materializes scalar subqueries and binds attributes to `BoundReference`s indexing `requiredSchema`. The conf is deliberately not rechecked at execution time. Once extraction has dropped a bloom from the post-scan `Filter`, the runtime has to honour that decision or fail, and it does the second.
**`FileFormat` API.**
- A new `buildReaderWithStorageFilters` overload takes `storageFilters: Seq[Expression]` and the metric map. Its default body requires `storageFilters` to be empty and otherwise delegates to `buildReaderWithPartitionValues`, so a format that does not implement the feature rejects a filter it cannot honour rather than dropping it.
- `FileSourceScanExec.inputRDD` only routes through the new entry point when there is something to push, so a `ParquetFileFormat` subclass that customizes reading by overriding `buildReaderWithPartitionValues` keeps working unchanged.
**Parquet reader.**
- A new `ParquetStorageFilter` value object holds the bound expressions, the key-column indices into the requested schema, and the optional metrics. `rewriteForMissingKeys` and `evalAllMissing` handle schema evolution, where a key column is in the requested schema but absent from the physical file. The rewritten predicate is evaluated against the value the reader will actually materialize for that column, which is the column's existence `DEFAULT` when it has one and null otherwise. Evaluating against null instead would filter on a value the scan never returns, and `XxHash64` is `nullable = false` so a null input hashes to the seed rather than producing null.
- `SpecificParquetRecordReaderBase` exposes the underlying `ParquetFileReader`, the input file and the footer, so the late-materialization driver can switch the requested schema per phase and call `readFilteredRowGroup(blockIdx, rowRanges)`.
- `VectorizedParquetRecordReader` gains a three-phase per-row-group loop driven by that one reader. Phase 0 computes `pushedFilterRanges` from the pushed data filter through the column index, which is metadata only. Phase 1 switches to the key-only schema, reads the key columns under those ranges, evaluates the storage filter per row, and accumulates the survivors into per-key-column queues of capacity-sized `WritableColumnVector`s. Phase 2 switches to the non-key schema and reads those columns under the surviving ranges. Emit splices the dequeued key vectors with the freshly read non-key vectors in the projection's own order.
- A projection that is all key columns skips phase 2 entirely and reconstructs every batch from the key queues, which is the shape with the largest saving.
- Phase 0 checks `parquet.filter.columnindex.enabled` itself, because `ParquetFileReader.getRowRanges` only asks whether a filter is pushed. That conf is the documented escape hatch for a file whose column index is wrong, and trusting a wrong column index here would drop rows for good.
- `requireOffsetIndexesForPhase2` fails at reader init if any projected column of any row group has no offset index. Phase 2 reads a strict subset of a row group's rows, which parquet can only do through the offset index, and neither widening the read nor skipping the filter is correct. The check reads only footer fields, and it covers every projected column rather than the non-key ones alone, because parquet builds one column index store per row group and returns an empty one as soon as any path in it lacks an offset index.
- Per-key-column value copying uses a `ValueCopier` chosen once at init, so the survivor loop has no per-value type dispatch.
- `initBatch` threads a `skipDataSlots` set through `allocateColumns`, so the persistent output vectors for key slots are never allocated. Under splicing those slots come from the queues.
- Preconditions that planning already guarantees throw instead of falling back, for the same reason the conf is not rechecked. That covers a key ordinal out of range, a non-primitive key column, missing key pages, and a vectorized-reader conf flipped between planning and execution. The two remaining silent fallbacks cannot change the result, namely a reader with no underlying `ParquetFileReader`, which only test mocks produce, and a file where every key column is missing, which is answered by evaluating the rewritten constant predicate.
**Metrics.** Five, all created only when the scan has storage filters, and all scoped to what the storage filter added on top of a no-storage-filter read of the same projection. Each counter names its own quantity, so the three verbs are deliberate. A row group is skipped, meaning its data columns were never read while phase 1 did read its key columns. A row is excluded, meaning it never reached the output. A byte is avoided, meaning it was never transferred.
- `storageFilterRowGroupsSkipped`, "row groups skipped by storage filter".
- `storageFilterRowsExcludedByRowGroup`, "rows excluded by storage filter (whole row group)".
- `storageFilterRowsExcludedWithinRowGroup`, "rows excluded by storage filter (within row group)". The suffix says where the row was excluded rather than by which mechanism, because a row sharing a page with a survivor is read and dropped during decode.
- `storageFilterBytesAvoidedByRowGroup`, "bytes avoided by storage filter (whole row group)".
- `storageFilterBytesAvoidedByPageFiltering`, "bytes avoided by storage filter (page filtering)".
The byte counters must not cost IO to report, so `compressedBytesForRowRanges` answers from the footer's `ColumnChunkMetaData.getTotalSize()` whenever the row range covers the whole block, and walks the offset index only for a strict subset. That is complete rather than a mitigation. A range narrower than the block can only come from column-index filtering, which builds and memoizes the store as a side effect, and phase 2's own read builds it before the walk in the other case. The walk counts the dictionary page too, since parquet reads it whenever it reads any data page of a chunk.
### Why are the changes needed?
A runtime bloom filter from join runtime filtering is applied as a post-scan `FilterExec` today. The scan still reads every value page of every row group, even where the bloom drops almost every row immediately. On a selective join over a wide table that read is the dominant cost.
Late materialization turns that around. The scan reads the bloom's key column first, decides which rows survive, and never reads the value pages no surviving row touches. A row group where nothing survives costs one key-column read and no value IO at all.
### Does this PR introduce _any_ user-facing change?
No change by default, since the conf is off and nothing is attached to a scan in that case.
With the conf on:
- Queries return the same rows. The late-materialization path is exact, and a filter that fails any eligibility check keeps its existing post-scan `FilterExec`.
- `FileSourceScanExec` reports the five metrics above in the SQL UI, and its description gains a `StorageFilters` entry.
- A scan reading a Parquet file written without offset indexes fails with an error naming the conf to turn off. Files written by parquet-mr 1.11 and later always have them.
- A consumer that illegally retains a `ColumnarBatch` across `next()` sees a sharper edge than before. On the plain path the previous batch's key vectors are reused, and under splicing they are freed at the next `nextBatch()`, so with off-heap vectors the retained reference points at released memory rather than stale values. The contract already forbids retaining a batch.
- Phase 1 buffers a whole row group's surviving key values before it produces that row group's first batch, so a task holds up to one extra copy of the key columns for one row group. The conf's documentation says so.
### How was this patch tested?
New `ParquetStorageFilterSuite`, 89 tests. Four blind review rounds ran over the change, each by an agent with no knowledge of the earlier ones, and every finding is fixed in this commit.
- Reader-level tests over hand-built filters: whole row group rejected, nothing rejected, mixed, multi-batch emit, key-only projection, a survivor count that is an exact multiple of the batch capacity, and a projection whose key column is not in the leading slot.
- One case per `ValueCopier` branch across 17 types and both encodings, each comparing the splicing path against the plain reader over the same file, so values are checked and not only row counts. `isSupportedKeyType` is asserted to cover exactly the copier's types.
- Nullable keys, two key columns, partition columns alongside spliced keys, off-heap vectors, the row-at-a-time path, `_metadata.row_index`, and a complex non-key column.
- Schema evolution: `rewriteForMissingKeys` and `evalAllMissing` as units, plus end-to-end reads where a key column is missing from the older file with a `DEFAULT`, with a `DEFAULT` that fails the filter, and with no `DEFAULT`.
- Page-level ranges, which need a multi-page row group: phase 1 stays aligned with the row indexes, and the byte metrics stay non-negative.
- Metric arithmetic: emitted plus excluded accounts for every row of the file, and an all-key projection reports zero avoided bytes.
- Planner gates: the bloom stays in the post-scan `Filter` when the vectorized reader is unavailable, when the conf is off, and when the conjunct is non-deterministic. A scan whose vectorized reader is disabled after planning fails loudly. Extraction preserves results with AQE on and off. Canonicalization keeps a storage-filter scan distinct from a plain one, so exchange and subquery reuse cannot cross them.
- Whole-stage codegen off, where the planner's gate is weaker than the runtime's, so the bloom is extracted and the spliced batch is served one row at a time.
- Column-index filtering off, the branch that decides where phase 0's ranges come from. The row accounting tells the two arms apart, since with the column index off every row of the block reaches phase 1.
- A limit under whole-stage codegen, which is the only shape that closes the spliced batch from outside while the reader is still open. The test asserts the generated source contains that close, so it cannot pass for the wrong reason.
- The byte metrics cost no extra IO, measured rather than argued. The same read runs twice over a filesystem that counts every byte handed back, once with all five metrics wired and once with none, and the counts match. Both range shapes are covered, the offset-index one and the footer one.
Knowingly untested, both for the same reason. The offset-index check's throw needs a file with no offset index, which parquet-mr cannot write. The `ParquetFileFormat`-subclass bypass needs a third-party subclass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code with Claude Opus 4.7 and Claude Opus 5
Co-authored-by: Matt Butrovich <mb...@gmail.com>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4149436424
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, TaskContext}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. None counts what it cost. A row group
+ * whose filter was given up reads its key columns twice and counts nothing, so the counters can
+ * show a saving on a scan that read more than it would have with the feature off.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums those skipped row groups' rows, counting per block the rows
+ * that survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned. A row group that gave
+ * splicing up reads its key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+
+ /** These counters keyed for the scan, which carries them without naming any of them. */
+ def toMap: Map[String, SQLMetric] = Map(
+ StorageFilterMetrics.ROW_GROUPS_SKIPPED -> rowGroupsSkipped,
+ StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP -> rowsExcludedByRowGroup,
+ StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP -> rowsExcludedWithinRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP -> bytesAvoidedByRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING -> bytesAvoidedByPageFiltering)
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /** A fresh set of counters, with the labels the SQL UI shows. */
+ def create(sparkContext: SparkContext): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped =
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ rowsExcludedByRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ rowsExcludedWithinRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ bytesAvoidedByRowGroup = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ bytesAvoidedByPageFiltering = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made: the map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer
+ // construction to first use on the executor.
+ @transient private lazy val predicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ require(boundExpression != null,
+ "a storage filter must only be evaluated on an executor, and this copy has released the " +
+ "expression it would build its predicate from")
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay.
+ */
+ def preparedPredicate: BasePredicate = predicate
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queues where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader evaluates
+ * the predicate without the conjuncts that precede it in the plan, so one that throws on a row an
+ * earlier conjunct would have rejected throws where a plain scan does not. That is handled where
+ * it arises. The reader gives the filter up for the row group and reads it plainly, and the
+ * post-scan `Filter` then evaluates every conjunct in its own order.
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows, as it is, an error the reader met while applying a storage filter that it must not
+ * absorb by giving the filter up. Two kinds are not the reader's to absorb, and both are asked
+ * the way the rest of Spark asks them:
+ * - a pending task kill, since the error may be the kill's interrupt surfacing through a UDF
+ * that wrapped it. `PythonRunner` reads a kill off the task context in the same way.
+ * - a fatal error in the cause chain, found by `Executor.isFatalError` as deep as the executor
+ * itself looks, which is `spark.executor.killOnFatalError.depth`.
+ *
+ * As it is, because a `TaskKilledException` of the reader's own would be read as a corrupt file
+ * under `ignoreCorruptFiles`.
+ */
+ def rethrowIfMustPropagate(e: Throwable): Unit = {
+ val context = TaskContext.get()
+ val killed = context != null && context.isInterrupted()
+ if (killed || Executor.isFatalError(e, fatalErrorDepth)) throw e
Review Comment:
Fixed both ways you suggest. `rethrowIfMustPropagate` now wraps a kill or a fatal error in a checked `SparkException`, which `shouldIgnoreCorruptFileException` rejects. The executor still reports a killed task as killed, and still finds the fatal cause, since the check looks one level less deep than `spark.executor.killOnFatalError.depth` to leave room for the wrapper. The scaladoc says why.
A survivor vector that cannot grow no longer reaches that path. The append has its own catch, which gives splicing up and releases the buffer, since the allocation is optional, and the filter stays applied.
The kill test now has the key expression mark the kill while it throws, with a checked and with an unchecked error, and a fatal arm puts an OOM inside an unchecked error, the way `reserve` reports one. Each arm asserts that `shouldIgnoreCorruptFileException` rejects what reaches the task. Rethrowing as it is fails it. The buffer's own catch has no test, since it needs a vector that cannot grow.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1947,6 +1947,57 @@ object SQLConf {
.checkValue(threshold => threshold >= 0, "The threshold must not be negative.")
.createWithDefault(10)
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED =
+ buildConf("spark.sql.parquet.storageFilterPushdown.enabled")
+ .doc("If true, the vectorized Parquet reader may apply a runtime storage filter, such as a " +
+ "bloom filter from join runtime filtering, while it reads. It reads the columns the " +
+ "filter needs first, evaluates the filter per row, and then reads the remaining columns " +
+ "only for the rows that survived. This is a planning-time decision. " +
+ "A filter that is attached also stays in the post-scan filter, the way a pushed data " +
+ "filter does, so the reader is free to stop applying it wherever doing so would cost " +
+ "more than it saves, and the answer does not change. A row group where it stops reads " +
+ "its key columns twice, which is slower than not pushing the filter. " +
+ "Under spark.sql.files.ignoreCorruptFiles the answer can change, because this reader " +
+ "reads different pages in a different order than a plain read. Which rows survive a " +
+ "corrupt page can then differ from a plain read, in either direction. " +
+ "Returning only the surviving rows of a row group relies on the Parquet page index, so a " +
Review Comment:
Restored, at the end of that paragraph. It says a read with pushed data filters already relies on the page index, and that conf turns it off there too. A read with no pushed data filter relies on it only with this feature. The description's user-facing section and its design section now name two exceptions, `ignoreCorruptFiles` and a wrong page index, with the same point.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
+ * read for the rest of this split. The pushed data filter's own page pruning still applies, since
+ * parquet builds those ranges and hands back whole any block it has no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a read that throws in between leaves them reachable for {@link #close()}.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key column.
+ List<Type> requestedFields = requestedSchema.getFields();
+ boolean anyNonKeyPresent = false;
+ for (int i = 0; i < requestedFields.size() && !anyNonKeyPresent; i++) {
+ anyNonKeyPresent = !keyTopLevelNames.contains(requestedFields.get(i).getName())
+ && !missingColumns.contains(parquetColumn.children().apply(i));
+ }
+ if (!anyNonKeyPresent) return;
+ nonKeyColumns = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
Review Comment:
Added. `ParquetStorageFilter.throwIfKilled` runs at the top of the row-group loop and of phase 1's chunk loop, and throws the same wrapped form as the rethrow, so `ignoreCorruptFiles` cannot swallow it. The `MissingOffsetIndexException` catch asks it first too, so an interrupt that parquet turned into that exception is not taken for a missing index.
A new test has the key expression mark the task killed inside phase 1's first chunk of a single row group, and asserts the read stops there. Removing the chunk check fails it.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
+ * read for the rest of this split. The pushed data filter's own page pruning still applies, since
+ * parquet builds those ranges and hands back whole any block it has no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a read that throws in between leaves them reachable for {@link #close()}.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key column.
+ List<Type> requestedFields = requestedSchema.getFields();
+ boolean anyNonKeyPresent = false;
+ for (int i = 0; i < requestedFields.size() && !anyNonKeyPresent; i++) {
+ anyNonKeyPresent = !keyTopLevelNames.contains(requestedFields.get(i).getName())
+ && !missingColumns.contains(parquetColumn.children().apply(i));
+ }
+ if (!anyNonKeyPresent) return;
+ nonKeyColumns = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped, with every projected byte of the rows the pushed data filter kept as
+ * avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Exception e) {
Review Comment:
Narrowed in all three places to a key page whose decoding raises an exception. Each also says that a codec reporting corruption as an `Error`, as hadoop-lzo does, fails before the row group's first batch. The description's `ignoreCorruptFiles` paragraph says the same.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
+ * read for the rest of this split. The pushed data filter's own page pruning still applies, since
+ * parquet builds those ranges and hands back whole any block it has no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a read that throws in between leaves them reachable for {@link #close()}.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key column.
+ List<Type> requestedFields = requestedSchema.getFields();
+ boolean anyNonKeyPresent = false;
+ for (int i = 0; i < requestedFields.size() && !anyNonKeyPresent; i++) {
Review Comment:
Fixed as you suggest. The decline asks whether any non-key leaf exists in the file schema, through `fileSchema.containsPath`. A new test reads a file holding `s STRUCT<a>` as `s STRUCT<b>` under `returnNullStructIfAllFieldsMissing=true`, with a filter that rejects every row, and asserts the file comes back whole. The old check fails it.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2757 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkException, TaskContext}
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading the
+ // rest of a file without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender("partial-read warning")
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val warned = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering"))
+ (result.get, warned)
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString),
+ useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(195L))
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
Review Comment:
Both done. The initialization sums into a local, so a second pass is harmless. The test now subclasses `VectorizedParquetRecordReader`, counts the `initializeInternal` calls behind the 2-arg `initialize`, and asserts one. Putting the second call back fails it. The description says that instead of the NPE.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
+ * read for the rest of this split. The pushed data filter's own page pruning still applies, since
+ * parquet builds those ranges and hands back whole any block it has no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a read that throws in between leaves them reachable for {@link #close()}.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key column.
+ List<Type> requestedFields = requestedSchema.getFields();
+ boolean anyNonKeyPresent = false;
+ for (int i = 0; i < requestedFields.size() && !anyNonKeyPresent; i++) {
+ anyNonKeyPresent = !keyTopLevelNames.contains(requestedFields.get(i).getName())
+ && !missingColumns.contains(parquetColumn.children().apply(i));
+ }
+ if (!anyNonKeyPresent) return;
+ nonKeyColumns = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped, with every projected byte of the rows the pushed data filter kept as
+ * avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Exception e) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. Giving the row group up hands it to phase 2 whole, which reads
+ // the same pages in the same batches as a plain scan. So the page fails at the same batch,
+ // after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain read.
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) splicedBytes += appendSurvivorRowToAccumulators(r);
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together
+ // here.
+ long rangeBytes = survivorRangeCount * ROW_RANGE_BYTES;
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ } catch (Exception e) {
+ // Fail open on any exception. The predicate runs here without the conjuncts before it in
+ // the plan, so it can throw on a row a plain scan never evaluates it on (see
+ // `ParquetStorageFilter.isSupportedStorageFilter`). Giving the filter up returns every
+ // row the pushed filter kept, and the post-scan Filter decides them in its own order.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK error
+ // (`timestamp_seconds` of a decimal), and a Hive UDF throws a checked SparkException.
+ // Rethrowing would fail a query a plain read answers, or drop the rest of the file
+ // silently under `ignoreCorruptFiles`. Only a task kill or a fatal error is rethrown,
+ // first. A failed copy of a surviving key lands here too, and gets the same answer.
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, ROW_GROUP_GIVEN_UP_ON_EVALUATION);
+ return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, and returns what that costs the
+ * budget up front. A variable-length key's byte child is sized to what the set queued before it
+ * held, so it does not grow from the default one doubling at a time for every capacity's worth
+ * of survivors. That memory is taken before any value arrives, so it is charged now, as if those
+ * values had arrived, and the values written into it use that charge up before they add to it.
+ */
+ private long allocateKeyAccumulators() {
+ WritableColumnVector[] previous = survivorBatches.peekLast();
+ // Assigned before the loop, so an allocation failure part way through leaves the vectors
+ // allocated so far reachable for `abandonSplicing`.
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ long charged = 0L;
+ for (int i = 0; i < keyColumns.length; i++) {
+ WritableColumnVector accumulator = newKeyVector(i);
+ currentKeyAccumulators[i] = accumulator;
+ prepaidValueBytes[i] = 0L;
+ if (keyColumns[i].variableLength() && previous != null) {
+ int valueBytes = previous[i].arrayData().getElementsAppended();
Review Comment:
Added. The test's uncapped arm also reads through `readAllWith` at capacity 4 and asserts every `(k, v)` pair of the six survivors, so the keys past the first accumulator come from the one sized up front.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5895124425
@dongjoon-hyun thank you. Round 5 is answered in the threads, and the description is up to date.
- A checked `SparkException` from a Hive UDF key fails open now, both per row and for the constant of a file missing its key column. Both cases have a test.
- The retry is simpler, as you proposed. The unreachable widening, the terminator overrides and `hitEndOfData` are gone, and the base class keeps master's `nextBatch` check.
- The dictionary-id memoization is a follow-up in the description.
- Everything else is fixed as you suggested.
Since this round reviewed the restructured reader, I am keeping that shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5798769693
Thank you for the quick update, @peter-toth. The review fixes addressed the previous round well.
I left another round of inline comments on `b9981cda93b` ([review](https://github.com/apache/spark/pull/58895#pullrequestreview-5293839352)). Here is the summary.
**Correctness (these seem worth addressing before merge)**
- **Evaluation order:** the extracted bloom is now evaluated before the remaining post-scan conjuncts. With ANSI mode, a key like `CAST(s AS BIGINT)` from `InjectRuntimeFilter` can throw on rows that an earlier guard used to filter out, so a query that works with the conf off fails with it on.
- **`ignoreCorruptFiles`:** the same evaluation errors now happen inside `FileScanRDD`'s catch, so they are treated as a corrupt file and the rest of the file is silently skipped.
- **Phase-2 memory:** per-row `finalRanges` make each non-key column reader build its own `RowRange` list. That can reach hundreds of MB per task, and `maxSplicedRowGroupBytes` doesn't bound it.
- **Misleading failure:** the per-block `ColumnIndexStore` is memoized over all columns, so a missing or unreadable offset index on one key column fails phase 2 with a "written without a page index" message.
**Minor**
- `bytesAvoidedByPageFiltering` over-reports, because `SQLMetric.add` drops negative values.
- The error message embeds the full bloom as hex.
- Metrics are not updated when an all-missing-key file is skipped.
**Design / cleanup**
- Since the runtime bloom is result-preserving, could it be an optional hint (fall back to the plain read) instead of must-honor with a new exception type?
- The all-keys reader branch is unreachable in production, which contradicts the description.
- Some things could be simplified: survivor-buffer memory accounting, reusing `RowToColumnConverter`, per-row `byte[]` copies, per-file bloom re-deserialization, the `FileFormat` default-builder recursion trap, and SQLConf placement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4135661859
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetBlockAccounting.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.column.columnindex.OffsetIndex;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+
+/**
+ * What a partial read of a Parquet file transfers, and whether one is possible at all, answered
+ * from the footer the {@link ParquetFileReader} already holds. Every method here is metadata
+ * arithmetic and causes no IO of its own, which is what lets a reader report these numbers per row
+ * group.
+ *
+ * <p>One instance per file, because the warning below is reported once per file.
+ */
+final class ParquetBlockAccounting {
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(ParquetBlockAccounting.class);
+
+ private final ParquetFileReader reader;
+
+ /** Whether the byte count has already been reported as undercounting on this file. */
+ private boolean loggedMissingStoreEntry;
+
+ ParquetBlockAccounting(ParquetFileReader reader) {
+ this.reader = reader;
+ }
+
+ /** Resolves a column list's leaf paths, so the per-row-group footer walks do not have to. */
+ static Set<ColumnPath> pathsOf(List<ColumnDescriptor> columns) {
+ Set<ColumnPath> paths = new HashSet<>();
+ for (ColumnDescriptor column : columns) {
+ paths.add(ColumnPath.get(column.getPath()));
+ }
+ return paths;
+ }
+
+ /**
+ * Whether every one of {@code paths} that this file has can be read in part, which needs an
+ * offset index for it in this block. A column the file does not have is no obstacle, since a read
+ * never reads it and it is not in the walk to begin with.
+ */
+ boolean hasOffsetIndexes(int blockIndex, Set<ColumnPath> paths) {
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ if (paths.contains(chunk.getPath()) && chunk.getOffsetIndexReference() == null) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Compressed bytes a reader transfers for the leaf columns named by {@code paths} when it reads
+ * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are
+ * included, since both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block, and the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters.
+ * Whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset, and the walk goes over the offset index, as
+ * parquet's own read path does, adding the dictionary page the way
+ * {@code calculateOffsetRanges} does. The store is guaranteed to exist here, so the walk is
+ * pure metadata arithmetic. For ranges a reader narrowed itself, which column-index
+ * filtering had no hand in, that guarantee is an ordering one, since the reader's own read
+ * of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct, since the reader transfers nothing for them. They fall out of the walk by themselves,
+ * which goes over the block's chunks rather than over {@code paths}.
+ */
+ long compressedBytesForRowRanges(
+ int blockIndex,
+ Set<ColumnPath> paths,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (paths.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ BlockMetaData block = reader.getRowGroups().get(blockIndex);
+ long blockRowCount = block.getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnChunkMetaData chunk : block.getColumns()) {
+ ColumnPath path = chunk.getPath();
+ if (!paths.contains(path)) {
+ // A leaf of the file this walk is not about, either not projected at all, or projected but
+ // read in another phase than the one being measured.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ // The store answers null, rather than throwing, for a path it was not built with. That can
+ // only mean the block's store was built while a narrower schema was requested than this
+ // walk asks about, an ordering bug rather than a property of the file, and the footer tells
+ // the two apart. It is reported rather than thrown, because this walk only produces a
+ // counter, and `ignoreCorruptFiles` turns any exception from a reader into a silently
+ // truncated file, so a byte metric must not be able to change the answer.
+ if (chunk.getOffsetIndexReference() != null && !loggedMissingStoreEntry) {
+ loggedMissingStoreEntry = true;
+ LOG.warn("Undercounting the storage filter's avoided bytes for {}: column {} of row "
+ + "group {} has an offset index the block's column index store was not built with",
+ MDC.of(LogKeys.PATH, reader.getFile()),
+ MDC.of(LogKeys.COLUMN_NAME, path.toDotString()),
+ MDC.of(LogKeys.INDEX, blockIndex));
+ }
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
Review Comment:
This measures the dictionary page as `[getStartingPos(), getFirstDataPageOffset())`, but parquet's read uses `[getStartingPos(), offsetIndex.getOffset(0))`, because `ParquetFileReader.internalReadFilteredRowGroup` passes `offsetIndex.getOffset(0)` to `calculateOffsetRanges`. The two differ when the footer's `data_page_offset` points at the dictionary page. parquet-mr 1.11.0-1.11.2 wrote files that way, with an offset index as well; PARQUET-1977 fixed it in 1.12.0.
For such files this returns 0, which skews the metrics:
- A partial walk misses the dictionary, while the whole-block baseline (`getTotalSize()`) includes it. So `bytesAvoidedByPageFiltering` is over-reported by the dictionary size for each dictionary-encoded column.
- `bytesAvoidedByRowGroup` is under-reported when a pushed data filter narrowed the baseline.
The offset index is already at hand in the loop above. Could the upper bound be `offsetIndex.getOffset(0)` (guarding `getPageCount() > 0`), as parquet does? That would also make the comment at L156 accurate.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -752,15 +774,36 @@ case class FileSourceScanExec(
lazy val inputRDD: RDD[InternalRow] = {
val options = relation.options +
(FileFormat.OPTION_RETURNING_BATCH -> supportsColumnar.toString)
- val readFile: (PartitionedFile) => Iterator[InternalRow] =
- relation.fileFormat.buildReaderWithPartitionValues(
+ val hadoopConf = getHadoopConf(relation.sparkSession, relation.options)
Review Comment:
Minor, latent: both builders now share one `Configuration`. Before, `getHadoopConf` gave the single builder call its own copy. A format whose `buildReaderWithStorageFilters` mutates the conf and then returns `None` would leak that setup into `buildReaderWithPartitionValues`; `ParquetFileFormat.buildParquetReader` does such a mutation with `setupHadoopConf`. `ParquetFileFormat` declines before mutating, so nothing in-tree hits this, but the trait doc doesn't forbid it. Passing a fresh `getHadoopConf(...)` to the fallback costs one copy on the driver.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled once the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Set to true once this reader has no rows left to return, which is when the row groups run out
+ * or when the filter rejects the whole file.
+ */
+ private boolean hitEndOfData = false;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch is a view whose slots alias these vectors (non-key and partition) and the head of
+ // each survivor queue (key slots), so closing it would re-close a shared vector and free the
+ // same buffer twice. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ @Override
+ public float getProgress() {
+ // Under a storage filter, rowsReturned counts survivors while totalRowCount is the count before
+ // the filter narrowed anything, so the ratio neither reaches 1 on its own nor stays below it. A
+ // row group read whole after the page index is given up can return more rows than that count.
+ // `hitEndOfData` is the real terminator here, and the ratio is clamped to what it means.
+ if (hitEndOfData) return 1.0f;
+ return Math.min(1.0f, super.getProgress());
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file. True, or an error, reads it the way a plain scan would, under the same
+ * fail-open rule as a row's value, since the constant can be one the predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (RuntimeException e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ hitEndOfData = true;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Whether this read has returned every row it is going to.
+ *
+ * <p>{@code totalRowCount} is how many rows the file yields before this filter narrows anything,
+ * so it is a terminator only while there is no filter. Under one it is an upper bound the read
+ * never reaches, and a row group read whole after the page index was given up can pass it. What
+ * ends such a read is {@link #hitEndOfData}, set when the row groups run out or when the filter
+ * rejects the whole file.
+ */
+ @Override
+ protected boolean noMoreRows() {
+ return hitEndOfData || (storageFilter == null && super.noMoreRows());
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
Review Comment:
Minor, latent: `initializeLateMaterialization()` is not idempotent, but it runs twice when this reader is initialized through Hadoop's 2-arg `initialize(InputSplit, TaskAttemptContext)`. `VectorizedParquetRecordReader`'s 2-arg override calls `super.initialize(split, ctx)`. That dispatches to the 5-arg override, which already calls `initializeInternal()`, and the 2-arg override then calls it again. If the first pass declined with `storageFilter = null` (the all-keys case at L560), the second pass gets past the `fileReader == null` guard and throws an NPE here instead of reading plainly.
The double call was harmless before, because `initializeInternal()` was private and idempotent. Nothing in-tree calls the 2-arg overload on this class today, so an early `if (storageFilter == null) return;` would be enough. Dropping the extra `initializeInternal()` from the 2-arg override would also work.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java:
##########
@@ -83,54 +79,45 @@ final class ParquetReadState {
* levels. */
boolean shouldSkip;
- ParquetReadState(
- ColumnDescriptor descriptor,
- boolean isRequired,
- PrimitiveIterator.OfLong rowIndexes) {
+ private ParquetReadState(ColumnDescriptor descriptor, boolean isRequired) {
this.maxRepetitionLevel = descriptor.getMaxRepetitionLevel();
this.maxDefinitionLevel = descriptor.getMaxDefinitionLevel();
this.isRequired = isRequired;
- this.rowRanges = constructRanges(rowIndexes);
- nextRange();
}
/**
- * Construct a list of row ranges from the given `rowIndexes`. For example, suppose the
- * `rowIndexes` are `[0, 1, 2, 4, 5, 7, 8, 9]`, it will be converted into 3 row ranges:
- * `[0-2], [4-5], [7-9]`.
+ * The state for reading {@code descriptor}, told which rows to include the best way the caller
+ * can say it.
+ *
+ * <p>{@code rowRanges} is the rows the read was asked for, when whoever asked knew them. A caller
+ * that filtered the rows itself holds their ranges, and every column reader of the row group can
+ * then walk that one list. Null says it does not, and the page store is left to describe its
+ * rows, which it can only do as {@code rowIndexes}, one index per row. No row indexes in turn
+ * means every row of the chunk is included.
*/
- private Iterator<RowRange> constructRanges(PrimitiveIterator.OfLong rowIndexes) {
- if (rowIndexes == null) {
- return null;
- }
-
- List<RowRange> rowRanges = new ArrayList<>();
- long currentStart = Long.MIN_VALUE;
- long previous = Long.MIN_VALUE;
-
- while (rowIndexes.hasNext()) {
- long idx = rowIndexes.nextLong();
- if (currentStart == Long.MIN_VALUE) {
- currentStart = idx;
- } else if (previous + 1 != idx) {
- RowRange range = new RowRange(currentStart, previous);
- rowRanges.add(range);
- currentStart = idx;
- }
- previous = idx;
- }
-
- if (previous != Long.MIN_VALUE) {
- rowRanges.add(new RowRange(currentStart, previous));
+ static ParquetReadState forRead(
+ ColumnDescriptor descriptor,
+ boolean isRequired,
+ RowRanges rowRanges,
+ PrimitiveIterator.OfLong rowIndexes) {
+ ParquetReadState state;
+ if (rowRanges != null) {
Review Comment:
`forRead` now prefers the caller's `rowRanges` over the store's `rowIndexes`. Before, the range state always came from the store that supplies the pages. So if rows were restricted, the pages carried a first-row index by construction. Now that holds only because `readerRangesFor` (`LateMaterializationParquetRecordReader.java:884`) returns `null` under exactly the condition of parquet's private whole-block fallback in `readFilteredRowGroup(int, RowRanges)` (`rowCount == block.getRowCount()` -> `internalReadRowGroup`).
If the two ever diverge, e.g. after a parquet upgrade or with another caller that pairs ranges with a store from `readRowGroup`, rows are misaligned with no error:
- `VectorizedColumnReader.readPage` sets `pageFirstRowIndex = page.getFirstRowIndex().orElse(0L)` on every page.
- `rowId` restarts at 0 on each page.
- With splicing, keys pair with another row's values.
Could the reader assert the pairing, e.g. `readerRanges == null || pages.getRowIndexes().isPresent()` before `installRowGroup` and before building the phase-1 readers, so a divergence fails loudly?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled once the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Set to true once this reader has no rows left to return, which is when the row groups run out
+ * or when the filter rejects the whole file.
+ */
+ private boolean hitEndOfData = false;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch is a view whose slots alias these vectors (non-key and partition) and the head of
+ // each survivor queue (key slots), so closing it would re-close a shared vector and free the
Review Comment:
On the rationale only, not reopening the swap discussion. Vector `close()` is idempotent:
- `OffHeapColumnVector.releaseMemory()` zeroes the addresses after freeing them.
- `OnHeapColumnVector` nulls its arrays.
- `WritableColumnVector.close()` nulls its children and dictionary ids.
This PR already depends on that. Under a LIMIT, the generated `ColumnarToRowExec` code closes the spliced batch, and the reader's `close()` then closes the same vectors, as the test comment at `ParquetStorageFilterSuite.scala:2518` says. So "free the same buffer twice" isn't what this avoids. What it needs is to reach the persistent key vectors that the splice array no longer holds.
The suite comment at L1620-1621 ("a double close ... is a crash") has the same issue. Off heap, a read after close is the real hazard. Could both comments say that instead?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -151,6 +151,66 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
}
}
+ /**
+ * The conjuncts of `afterScanFilters` the file format can evaluate at the storage layer for late
+ * materialization, to prune the IO of the columns the filter does not reference.
+ *
+ * They stay in the post-scan `Filter` as well, the way a pushed data filter does. The reader is
+ * offered them, not obliged to honor them, so the plan keeps the exact check. What it costs is
+ * evaluating the conjunct a second time for the rows the scan emits. What it buys is a reader
+ * free to give up on a file it cannot read in part, or on a stretch of rows whose survivors cost
+ * too much to keep track of, without the answer depending on it.
+ *
+ * Two of the conditions are per scan, and failing either offers nothing:
+ * - [[FileFormat.supportsStorageFilterPushdown]] holds. That is where a format reads the conf
+ * that enables this, so a format's own conf never decides for another format, and asking it
+ * first keeps everything below off the path of a scan that will not use it.
+ * - [[FileFormat.supportBatch]] holds for the schema the reader will see,
+ * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives
+ * its own vectorized-read decision from. Late materialization needs a batch read, so this asks
+ * about batch support rather than naming a format.
+ *
+ * The rest are per conjunct:
+ * - It is deterministic. The reader drops the rows the conjunct rejects and the post-scan
+ * `Filter` evaluates it again on the rows it keeps, so the two evaluations have to agree. Such
+ * a conjunct would also fail at task time, since a reader evaluates the predicate without
Review Comment:
Nit: "Such a conjunct would also fail at task time" no longer holds with the reader's fail-open catch. An uninitialized `Nondeterministic` predicate throws either `IllegalArgumentException` (`require(initialized, ...)` in `Nondeterministic.eval`) or an NPE from the unset RNG in generated code. Both are `RuntimeException`s, which `LateMaterializationParquetRecordReader` swallows. It would give the filter up on every row group, with a WARN per file.
The deterministic check is still needed for the reason in the first sentence; only this sentence is off. The test comment at `ParquetStorageFilterSuite.scala:1209` also refers to `ParquetStorageFilter.test`, which is now `preparedPredicate`.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled once the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Set to true once this reader has no rows left to return, which is when the row groups run out
+ * or when the filter rejects the whole file.
+ */
+ private boolean hitEndOfData = false;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch is a view whose slots alias these vectors (non-key and partition) and the head of
+ // each survivor queue (key slots), so closing it would re-close a shared vector and free the
+ // same buffer twice. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ @Override
+ public float getProgress() {
+ // Under a storage filter, rowsReturned counts survivors while totalRowCount is the count before
+ // the filter narrowed anything, so the ratio neither reaches 1 on its own nor stays below it. A
+ // row group read whole after the page index is given up can return more rows than that count.
+ // `hitEndOfData` is the real terminator here, and the ratio is clamped to what it means.
+ if (hitEndOfData) return 1.0f;
+ return Math.min(1.0f, super.getProgress());
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file. True, or an error, reads it the way a plain scan would, under the same
+ * fail-open rule as a row's value, since the constant can be one the predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (RuntimeException e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ hitEndOfData = true;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Whether this read has returned every row it is going to.
+ *
+ * <p>{@code totalRowCount} is how many rows the file yields before this filter narrows anything,
+ * so it is a terminator only while there is no filter. Under one it is an upper bound the read
+ * never reaches, and a row group read whole after the page index was given up can pass it. What
+ * ends such a read is {@link #hitEndOfData}, set when the row groups run out or when the filter
+ * rejects the whole file.
+ */
+ @Override
+ protected boolean noMoreRows() {
+ return hitEndOfData || (storageFilter == null && super.noMoreRows());
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
Review Comment:
This fallback can't do what the comment says. `readFilteredRowGroup(int, RowRanges)` returns `null` only for a 0-row block or empty ranges. Neither depends on the requested schema, and both are excluded above (L639, L656). If it ever did return `null` here, `survivors == null` leaves `finalRanges = pushedFilterRanges`. L748 then makes the same call with the same block and ranges, gets `null` again, and throws the `IllegalStateException` at L783, whose comment ("Match phase 1 and fail with a message") contradicts this branch giving up. Could both sites use the same assertion?
Relatedly, the comment on the empty-block guard (L640-641) says parquet's own `getRowRanges` builds `Range(0, -1)`, but parquet 1.18.1 returns `RowRanges.EMPTY` for a 0-row block. What would trip the `from <= to` assertion is this reader's own `RowRanges.createSingle(0)` in `pushedFilterRangesFor` under `pageIndexDisabled`. Naming that would keep the guard from looking removable.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled once the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Set to true once this reader has no rows left to return, which is when the row groups run out
+ * or when the filter rejects the whole file.
+ */
+ private boolean hitEndOfData = false;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch is a view whose slots alias these vectors (non-key and partition) and the head of
+ // each survivor queue (key slots), so closing it would re-close a shared vector and free the
+ // same buffer twice. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ @Override
+ public float getProgress() {
+ // Under a storage filter, rowsReturned counts survivors while totalRowCount is the count before
+ // the filter narrowed anything, so the ratio neither reaches 1 on its own nor stays below it. A
+ // row group read whole after the page index is given up can return more rows than that count.
+ // `hitEndOfData` is the real terminator here, and the ratio is clamped to what it means.
+ if (hitEndOfData) return 1.0f;
+ return Math.min(1.0f, super.getProgress());
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file. True, or an error, reads it the way a plain scan would, under the same
+ * fail-open rule as a row's value, since the constant can be one the predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (RuntimeException e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ hitEndOfData = true;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Whether this read has returned every row it is going to.
+ *
+ * <p>{@code totalRowCount} is how many rows the file yields before this filter narrows anything,
+ * so it is a terminator only while there is no filter. Under one it is an upper bound the read
+ * never reaches, and a row group read whole after the page index was given up can pass it. What
+ * ends such a read is {@link #hitEndOfData}, set when the row groups run out or when the filter
+ * rejects the whole file.
+ */
+ @Override
+ protected boolean noMoreRows() {
+ return hitEndOfData || (storageFilter == null && super.noMoreRows());
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Should parquet ever say
+ // otherwise, the filter is given up for this row group, which reads what a plain read
+ // reads. The alternative is to fail a query a plain read answers.
+ abandonSplicing();
+ survivors = null;
+ } else {
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ if (baselineRows != blockRowCount) {
Review Comment:
This widening looks unreachable with parquet 1.18.1. `MissingOffsetIndexException` comes only from the `EMPTY` `ColumnIndexStore`, and for that store `ColumnIndexFilter.calculateRowRanges` has already returned `createSingle(rowCount)`, either at initialize through `getFilteredRecordCount()` or in phase 0. So `baselineRows == blockRowCount` whenever this catch runs, and both arms read the whole block.
Even so, `noMoreRows()` (L387), the `getProgress()` clamp and the comments at L300-303 and L380-381 all compensate for the "more rows than `totalRowCount`" case it could produce. `rowsReturned <= totalRowCount` holds on every path. Could the base terminator stay, with the skipped file expressed as `totalRowCount = 0` (or `nextBlockIndex = blocks.size()`), and this branch dropped?
That would also stop `storageFilter == null` from doubling as a mode switch. The null routes `loadNextRowGroup` to `readNextFilteredRowGroup`, whose `currentBlock` cursor `readFilteredRowGroup(blockIdx, ...)` never advances. The javadoc at L105 invites nulling it after the first row group ("Nulled once the reader gives it up for the whole file"), and doing so would restart the file at block 0.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled once the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Set to true once this reader has no rows left to return, which is when the row groups run out
+ * or when the filter rejects the whole file.
+ */
+ private boolean hitEndOfData = false;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch is a view whose slots alias these vectors (non-key and partition) and the head of
+ // each survivor queue (key slots), so closing it would re-close a shared vector and free the
+ // same buffer twice. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ @Override
+ public float getProgress() {
+ // Under a storage filter, rowsReturned counts survivors while totalRowCount is the count before
+ // the filter narrowed anything, so the ratio neither reaches 1 on its own nor stays below it. A
+ // row group read whole after the page index is given up can return more rows than that count.
+ // `hitEndOfData` is the real terminator here, and the ratio is clamped to what it means.
+ if (hitEndOfData) return 1.0f;
+ return Math.min(1.0f, super.getProgress());
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file. True, or an error, reads it the way a plain scan would, under the same
+ * fail-open rule as a row's value, since the constant can be one the predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (RuntimeException e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ hitEndOfData = true;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Whether this read has returned every row it is going to.
+ *
+ * <p>{@code totalRowCount} is how many rows the file yields before this filter narrows anything,
+ * so it is a terminator only while there is no filter. Under one it is an upper bound the read
+ * never reaches, and a row group read whole after the page index was given up can pass it. What
+ * ends such a read is {@link #hitEndOfData}, set when the row groups run out or when the filter
+ * rejects the whole file.
+ */
+ @Override
+ protected boolean noMoreRows() {
+ return hitEndOfData || (storageFilter == null && super.noMoreRows());
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Should parquet ever say
+ // otherwise, the filter is given up for this row group, which reads what a plain read
+ // reads. The alternative is to fail a query a plain read answers.
+ abandonSplicing();
+ survivors = null;
+ } else {
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ if (baselineRows != blockRowCount) {
+ // The retry avoids the same wall because a store missing one column's offset index
+ // reports no column index either, so these ranges cover the whole block and parquet reads
+ // it without consulting an index. That is three parquet internals deep, so it is not
+ // assumed, and if they are narrower the block is read whole instead, which is a superset
+ // of them. The extra rows cost the post-scan Filter work, where failing here would fail a
+ // query a plain read answers, and under `ignoreCorruptFiles` would be read as a corrupt
+ // file and drop the rest of this one.
+ finalRanges = RowRanges.createSingle(blockRowCount);
+ finalRowCount = blockRowCount;
+ }
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ long keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up, since it read what a plain
+ // scan reads and the saving is a certain zero.
+ if (!filterGivenUp) {
+ // The same ranges as the baseline when nothing was pruned, and then the walk would arrive
+ // at the same number a second time.
+ long phase2Bytes = finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += blockAccounting.compressedBytesForRowRanges(blockIdx,
+ keyOnlyPaths, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ long filteredRows = baselineRows - keptRows;
+ if (filteredRows > 0) m.rowsExcludedWithinRowGroup().add(filteredRows);
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ RowRanges readerRanges = readerRangesFor(finalRanges, finalRowCount, blockRowCount);
+ installRowGroup(dataPages, readerRanges, keptRows);
+ return true;
+ }
+ hitEndOfData = true;
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queues at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ return spliceCurrentRowGroup && isKeyTopLevel[slot];
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses, since the rows its column index already excluded were never this filter's
+ // to save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ *
+ * <p>A file this reader can no longer read in part still comes through here, since these are the
+ * rows a plain read of the same query would read, and leaving them to the post-scan Filter
+ * instead would make the scan slower than having no storage filter at all.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * What the surviving rows of the current row group cost to hold as row ranges, which is the half
+ * of the budget that a row group buffering nothing still pays. A filter whose survivors are
+ * scattered makes one range per surviving row, and phase 2 needs the whole set to select its
+ * pages. One set is held, not one per column reader, and the readers of a row group walk that
+ * same list, each with a cursor of its own.
+ */
+ private static long rowRangeStateBytes(long rangeCount) {
+ // Parquet's own `RowRanges.Range` holds two longs, plus its object header and a list slot.
+ return rangeCount * 40L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group. The
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ ensureKeyScratchAllocated();
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below, which is the bytes buffered
+ // for splicing and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so whether anything survives
+ // at all was the only thing left to learn, and it does. Stop here, since the ranges are
+ // not needed and the rest of the key column need not be decoded or evaluated. The
+ // row groups the filter empties are still skipped, which is the case that returns a
+ // non-null empty range set below.
+ abandonSplicing();
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) splicedBytes += appendSurvivorRowToAccumulators(r);
+ // Both halves of what this row group retains grow per survivor, and either can cross
+ // the budget on its own, so they are weighed together here and nowhere else. The cheaper
+ // concession comes first, which is to release the buffer, and the filter goes too if the
+ // ranges alone still do not fit.
+ long rangeBytes = rowRangeStateBytes(survivorRangeCount);
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper concession first, releasing the buffer, which is a no-op for a row group
+ // that has already given splicing up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ if (rangeBytes > cap) {
+ // The ranges being built are about to be thrown away, so stop evaluating the rest.
+ return null;
+ }
+ }
+ } catch (RuntimeException e) {
Review Comment:
**This still misses a checked exception.** `catch (RuntimeException e)` here and at L365 does not catch `org.apache.spark.SparkException`, which extends `Exception`, and Spark's UDF wrappers throw it from Scala code without declaring it. `HiveUDFEvaluatorBase.evaluate()` wraps any `Throwable` from a Hive UDF into `QueryExecutionErrors.failedExecuteUserDefinedFunctionError`, which returns `new SparkException(errorClass = "FAILED_EXECUTE_UDF", ...)`, and `ScalaUDF` throws the same error.
A Hive UDF join key can reach this path. `InjectRuntimeFilter.isSimpleExpression` excludes `SCALA_UDF` and `PYTHON_UDF` but not Hive UDFs, which have no tree pattern and are usually deterministic, and `isSupportedStorageFilter` checks only the top-level node and the key types. For example:
```sql
SELECT f.v, d.y FROM fact f JOIN dim d ON my_hive_udf(f.a) = d.k
WHERE f.a IS NOT NULL AND d.y = 'x'
```
If `my_hive_udf` throws on `null`, phase 1 evaluates it on null rows, since pushed filters prune only at page granularity. The post-scan `Filter` drops those rows with `f.a IS NOT NULL` before it evaluates the bloom. The `SparkException` passes this catch and fails the task with `FAILED_READ_FILE`, while the same query succeeds with the feature off. `ignoreCorruptFiles` does not change that, because `shouldIgnoreCorruptFileException` matches only `RuntimeException | IOException | InternalError`.
Could both catches take `Exception`, with `logFilterGivenUpOnError` widened to match? Nothing inside the `try` declares a checked exception. The current fail-open tests only use `ArithmeticException` and ANSI cast errors, so a test with a throwing UDF key would pin this.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1947,6 +1947,48 @@ object SQLConf {
.checkValue(threshold => threshold >= 0, "The threshold must not be negative.")
.createWithDefault(10)
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED =
+ buildConf("spark.sql.parquet.storageFilterPushdown.enabled")
+ .doc("If true, the vectorized Parquet reader may apply a runtime storage filter, such as a " +
+ "bloom filter from join runtime filtering, while it reads. It reads the columns the " +
+ "filter needs first, evaluates the filter per row, and then reads the remaining columns " +
+ "only for the rows that survived. This is a planning-time decision. With it false, no " +
+ "storage filter is attached to a scan at all. A filter that is attached also stays in " +
+ "the post-scan filter, the way a pushed data filter does, so the reader is free to stop " +
+ "applying it wherever doing so would cost more than it saves, and the answer does not " +
+ "change. Narrowing a row group to part of its rows trusts the Parquet page index, which " +
+ "a read with no pushed data filter never consults, so on a file whose page index is " +
+ "wrong this can pair a row's key with another row's values. Setting " +
+ "parquet.filter.columnindex.enabled to false is the escape hatch, and it leaves the " +
+ "filter with the row groups it empties, which need no index at all. Note that the " +
+ "surviving key values of a whole row group are buffered before that row group's first " +
+ "batch is produced, so a task holds up to one extra copy of the key columns for one row " +
+ "group.")
+ .version("5.0.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES =
+ buildConf("spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes")
+ .internal()
+ .doc("The maximum memory, in bytes, that the vectorized Parquet reader holds for one row " +
+ "group while it applies a storage filter. Two things count against it, and both grow " +
+ "with the number of surviving rows. One is the key values buffered to splice into the " +
+ "output batches, which follow the reader's memory mode and so can be off heap. The other " +
+ "is the row ranges those rows fall into, always on heap, which the second phase needs to " +
+ "select its pages. The count is examined after every surviving row. Past the limit the " +
+ "reader releases the buffer and reads every projected column of the surviving rows " +
+ "instead, which costs one extra read of the key columns. Past it again it reads the row " +
+ "group with no filter applied at all, which is correct but as slow as not pushing the " +
Review Comment:
"as slow as not pushing the filter" undersells it. By the time the ranges pass the budget (`LateMaterializationParquetRecordReader.java:977-979`), phase 1 has already transferred and decoded the key chunks, and phase 2 then reads the whole projection, key columns included. So this path is strictly slower than not pushing the filter.
The same is true of every give-up path: evaluation error, no offset index, the columnindex escape hatch, and the `MissingOffsetIndexException` latch. The description says so ("plus one more read of its key columns"), but the shipped text doesn't:
- this sentence;
- the `enabled` doc (L1961-1963). It recommends `parquet.filter.columnindex.enabled=false` without saying that every row group it doesn't empty then reads its key columns twice, which with unclustered keys means more IO than turning the conf off;
- `LateMaterializationParquetRecordReader.java:788-789` ("the saving is a certain zero").
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.SparkContext
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * added on top of a read of the same projection without one.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums those skipped row groups' rows, counting per block the rows
+ * that survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it, and it is zero on an
+ * all-keys projection, which can avoid nothing.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page. An all-keys projection has no page filtering at
+ * all, and [[rowsExcludedWithinRowGroup]] still counts every row the filter dropped.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+
+ /** These counters keyed for the scan, which carries them without naming any of them. */
+ def toMap: Map[String, SQLMetric] = Map(
+ StorageFilterMetrics.ROW_GROUPS_SKIPPED -> rowGroupsSkipped,
+ StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP -> rowsExcludedByRowGroup,
+ StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP -> rowsExcludedWithinRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP -> bytesAvoidedByRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING -> bytesAvoidedByPageFiltering)
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /** A fresh set of counters, with the labels the SQL UI shows. */
+ def create(sparkContext: SparkContext): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped =
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ rowsExcludedByRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ rowsExcludedWithinRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ bytesAvoidedByRowGroup = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ bytesAvoidedByPageFiltering = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made: the map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer
+ // construction to first use on the executor.
+ @transient private lazy val predicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ require(boundExpression != null,
+ "a storage filter must only be evaluated on an executor, and this copy has released the " +
+ "expression it would build its predicate from")
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets the task reclaim those bytes for
Review Comment:
This holds on the columnar path, not on the row path, which is taken when whole-stage codegen is off or the schema is wider than `spark.sql.codegen.maxFields` (the planner checks only `supportBatch`). On that path the bytes stay alive:
- `FileSourceScanExec.doExecute`'s closure calls `UnsafeProjection.create(schema)` and so captures the scan node.
- The scan node's non-transient `storageFilters` hold the executed `ScalarSubquery`.
- That subquery's `result` is the same `Array[Byte]` this `Literal` wraps.
So nulling `boundExpression` doesn't release the bloom bytes (up to 8MB) for that task. Maybe qualify the comment, or keep this in mind for the broadcast item in the description.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled once the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Set to true once this reader has no rows left to return, which is when the row groups run out
+ * or when the filter rejects the whole file.
+ */
+ private boolean hitEndOfData = false;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch is a view whose slots alias these vectors (non-key and partition) and the head of
+ // each survivor queue (key slots), so closing it would re-close a shared vector and free the
+ // same buffer twice. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ @Override
+ public float getProgress() {
+ // Under a storage filter, rowsReturned counts survivors while totalRowCount is the count before
+ // the filter narrowed anything, so the ratio neither reaches 1 on its own nor stays below it. A
+ // row group read whole after the page index is given up can return more rows than that count.
+ // `hitEndOfData` is the real terminator here, and the ratio is clamped to what it means.
+ if (hitEndOfData) return 1.0f;
+ return Math.min(1.0f, super.getProgress());
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file. True, or an error, reads it the way a plain scan would, under the same
+ * fail-open rule as a row's value, since the constant can be one the predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (RuntimeException e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ hitEndOfData = true;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Whether this read has returned every row it is going to.
+ *
+ * <p>{@code totalRowCount} is how many rows the file yields before this filter narrows anything,
+ * so it is a terminator only while there is no filter. Under one it is an upper bound the read
+ * never reaches, and a row group read whole after the page index was given up can pass it. What
+ * ends such a read is {@link #hitEndOfData}, set when the row groups run out or when the filter
+ * rejects the whole file.
+ */
+ @Override
+ protected boolean noMoreRows() {
+ return hitEndOfData || (storageFilter == null && super.noMoreRows());
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Should parquet ever say
+ // otherwise, the filter is given up for this row group, which reads what a plain read
+ // reads. The alternative is to fail a query a plain read answers.
+ abandonSplicing();
+ survivors = null;
+ } else {
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ if (baselineRows != blockRowCount) {
+ // The retry avoids the same wall because a store missing one column's offset index
+ // reports no column index either, so these ranges cover the whole block and parquet reads
+ // it without consulting an index. That is three parquet internals deep, so it is not
+ // assumed, and if they are narrower the block is read whole instead, which is a superset
+ // of them. The extra rows cost the post-scan Filter work, where failing here would fail a
+ // query a plain read answers, and under `ignoreCorruptFiles` would be read as a corrupt
+ // file and drop the rest of this one.
+ finalRanges = RowRanges.createSingle(blockRowCount);
+ finalRowCount = blockRowCount;
+ }
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ long keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up, since it read what a plain
+ // scan reads and the saving is a certain zero.
+ if (!filterGivenUp) {
+ // The same ranges as the baseline when nothing was pruned, and then the walk would arrive
+ // at the same number a second time.
+ long phase2Bytes = finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += blockAccounting.compressedBytesForRowRanges(blockIdx,
+ keyOnlyPaths, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ long filteredRows = baselineRows - keptRows;
+ if (filteredRows > 0) m.rowsExcludedWithinRowGroup().add(filteredRows);
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ RowRanges readerRanges = readerRangesFor(finalRanges, finalRowCount, blockRowCount);
+ installRowGroup(dataPages, readerRanges, keptRows);
+ return true;
+ }
+ hitEndOfData = true;
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queues at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ return spliceCurrentRowGroup && isKeyTopLevel[slot];
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses, since the rows its column index already excluded were never this filter's
+ // to save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ *
+ * <p>A file this reader can no longer read in part still comes through here, since these are the
+ * rows a plain read of the same query would read, and leaving them to the post-scan Filter
+ * instead would make the scan slower than having no storage filter at all.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * What the surviving rows of the current row group cost to hold as row ranges, which is the half
+ * of the budget that a row group buffering nothing still pays. A filter whose survivors are
+ * scattered makes one range per surviving row, and phase 2 needs the whole set to select its
+ * pages. One set is held, not one per column reader, and the readers of a row group walk that
+ * same list, each with a cursor of its own.
+ */
+ private static long rowRangeStateBytes(long rangeCount) {
+ // Parquet's own `RowRanges.Range` holds two longs, plus its object header and a list slot.
+ return rangeCount * 40L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group. The
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ ensureKeyScratchAllocated();
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below, which is the bytes buffered
+ // for splicing and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
Review Comment:
An optimization idea for dictionary-encoded keys. When the single key column comes back with lazy dictionary decoding (`keyScratchVectors[0].hasDictionary()`), this still evaluates the predicate once per row. Dictionary decode, `XxHash64` and the bloom probes then repeat for every repeated key value.
A per-row-group verdict array indexed by `getDictionaryIds().getDictId(r)` (unknown/true/false, with null evaluated once) would evaluate at most D times instead of N. For example, a 1M-row row group with ~20K distinct dictionary-encoded keys would do at most 20K evaluations plus 1M array lookups.
Notes on the approach:
- It relies on the same invariant as the lazy getters.
- `hasDictionary()` needs re-checking per batch, because `readBatch` decodes the ids when a later page is plain.
- Fail-open is unchanged, since the first row of each id throws where it does today.
- This is separate from the "evaluate over the key vector" item in the description, which is about the cost of each evaluation rather than how many there are.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ValueCopier.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.*;
+
+/**
+ * Copies one key value between column vectors. Picked per key column at init time by
+ * {@link #forType(DataType)} and called per surviving row; the caller handles null sources.
+ *
+ * <p>Nothing here is about Parquet. A copier is chosen from the Spark type alone, and both sides of
+ * the copy are {@link WritableColumnVector}s.
+ *
+ * <p>The types this handles are also the types a storage filter's key columns may have:
+ * {@code ParquetStorageFilter.isSupportedKeyType} answers from {@link #supports}, so the planner's
+ * gate, the filter's own checks and the copy itself read one list. That list is narrower than
Review Comment:
There is a third exclusion: `TimestampNTZNanosType` and `TimestampLTZNanosType` (behind `spark.sql.timestampNanosTypes.enabled`). They are `AtomicType`s, `XxHash64` hashes them, the vectorized reader reads them in batches, and `InjectRuntimeFilter` has no type gate. So a bloom on a nanos join key is built but never offered here, which is safe but silent.
They use a two-child layout, so the long branch wouldn't do. Could the javadoc name them? Alternatively, they could get dedicated copiers (`putTimestampNTZNanos(row, src.getTimestampNTZNanos(row))` and the LTZ pair), with the per-row byte charge adjusted.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -80,23 +85,23 @@ public class VectorizedParquetRecordReader extends SpecificParquetRecordReaderBa
* Encapsulate writable column vectors with other Parquet related info such as
* repetition / definition levels.
*/
- private ParquetColumnVector[] columnVectors;
+ protected ParquetColumnVector[] columnVectors;
/**
* The number of rows that have been returned.
*/
- private long rowsReturned;
+ protected long rowsReturned;
Review Comment:
Nit: `rowsReturned`, `totalCountLoadedSoFar`, `rowIndexGenerator` and `initColumnReader` don't need to be protected. `LateMaterializationParquetRecordReader` never uses them (`rowsReturned` appears there only in a comment). So the class javadoc ("The protected members below are what it reaches and what it overrides") isn't accurate for these four. Could they stay private?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
Review Comment:
Yes, please. `spark.sql.parquet.storageFilterPushdown.enabled` says the answer does not change. Under `ignoreCorruptFiles`, a corrupt key page now drops the rows of its row group that a plain read returns before reaching that page, so one sentence there would keep the doc accurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4145453440
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled when the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ *
+ * <p>That happens only before the first row group is loaded. The plain path reads through
+ * parquet's own row-group cursor, and the phases below read row groups by index, which never
+ * advances it. So nulling this later would restart the read at the first row group.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch's key slots hold the head of each survivor queue, not the persistent key vectors, so
+ // closing that batch would leave those unreached. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, and the plain read path skips them too. Past here one
+ // would reach `pushedFilterRangesFor`, whose `RowRanges.createSingle(0)` under
+ // `pageIndexDisabled` builds Range(0, -1) and trips its `from <= to` assertion.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
Review Comment:
Both comments fixed. The narrowing comment says that with a pushed filter the store is built at initialize by `getFilteredRecordCount()`, over the whole projection. Phase 0's comment says the full projection goes back on first because `getRowRanges` builds an unseen block's store over the current paths. The whole-file skip's comment says its ranges are a cache hit, since `getFilteredRecordCount()` resolved every block's ranges at initialize.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled when the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ *
+ * <p>That happens only before the first row group is loaded. The plain path reads through
+ * parquet's own row-group cursor, and the phases below read row groups by index, which never
+ * advances it. So nulling this later would restart the read at the first row group.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch's key slots hold the head of each survivor queue, not the persistent key vectors, so
+ // closing that batch would leave those unreached. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, and the plain read path skips them too. Past here one
+ // would reach `pushedFilterRangesFor`, whose `RowRanges.createSingle(0)` under
+ // `pageIndexDisabled` builds Range(0, -1) and trips its `from <= to` assertion.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
Review Comment:
Fixed. The comment says it is closed at the end of the phase that reads it, since parquet closes no store it hands out. The field javadoc was the accurate one and stays.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled when the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ *
+ * <p>That happens only before the first row group is loaded. The plain path reads through
+ * parquet's own row-group cursor, and the phases below read row groups by index, which never
+ * advances it. So nulling this later would restart the read at the first row group.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch's key slots hold the head of each survivor queue, not the persistent key vectors, so
+ // closing that batch would leave those unreached. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, and the plain read path skips them too. Past here one
+ // would reach `pushedFilterRangesFor`, whose `RowRanges.createSingle(0)` under
+ // `pageIndexDisabled` builds Range(0, -1) and trips its `from <= to` assertion.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Neither depends on the columns
+ // requested, so giving the filter up would only meet the same null in phase 2.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ long keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // The same ranges as the baseline when nothing was pruned, and then the walk would arrive
+ // at the same number a second time.
+ long phase2Bytes = finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += blockAccounting.compressedBytesForRowRanges(blockIdx,
+ keyOnlyPaths, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ long filteredRows = baselineRows - keptRows;
+ if (filteredRows > 0) m.rowsExcludedWithinRowGroup().add(filteredRows);
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ RowRanges readerRanges = readerRangesFor(finalRanges, finalRowCount, blockRowCount);
+ installRowGroup(dataPages, readerRanges, keptRows);
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queues at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ return spliceCurrentRowGroup && isKeyTopLevel[slot];
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses, since the rows its column index already excluded were never this filter's
+ // to save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ *
+ * <p>A file this reader can no longer read in part still comes through here, since these are the
+ * rows a plain read of the same query would read, and leaving them to the post-scan Filter
+ * instead would make the scan slower than having no storage filter at all.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * What the surviving rows of the current row group cost to hold as row ranges, which is the half
+ * of the budget that a row group buffering nothing still pays. A filter whose survivors are
+ * scattered makes one range per surviving row, and phase 2 needs the whole set to select its
+ * pages. One set is held, not one per column reader, and the readers of a row group walk that
+ * same list, each with a cursor of its own.
+ */
+ private static long rowRangeStateBytes(long rangeCount) {
+ // Parquet's own `RowRanges.Range` holds two longs, plus its object header and a list slot.
+ return rangeCount * 40L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group. The
Review Comment:
Both rewritten. `evaluateStorageFilter` lists every null return: the first survivor of a row group that cannot be narrowed, the ranges passing the budget, and an error decoding the key pages or applying the filter. The class javadoc lists the whole-file cases: every key missing, no projected non-key column in the file, and `parquet.filter.columnindex.enabled=false`. It also says a row group without offset indexes, and the rest of a split after a failed partial read, keep only the row groups they empty.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled when the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ *
+ * <p>That happens only before the first row group is loaded. The plain path reads through
+ * parquet's own row-group cursor, and the phases below read row groups by index, which never
+ * advances it. So nulling this later would restart the read at the first row group.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch's key slots hold the head of each survivor queue, not the persistent key vectors, so
+ // closing that batch would leave those unreached. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, and the plain read path skips them too. Past here one
+ // would reach `pushedFilterRangesFor`, whose `RowRanges.createSingle(0)` under
+ // `pageIndexDisabled` builds Range(0, -1) and trips its `from <= to` assertion.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Neither depends on the columns
+ // requested, so giving the filter up would only meet the same null in phase 2.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ long keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // The same ranges as the baseline when nothing was pruned, and then the walk would arrive
+ // at the same number a second time.
+ long phase2Bytes = finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += blockAccounting.compressedBytesForRowRanges(blockIdx,
+ keyOnlyPaths, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ long filteredRows = baselineRows - keptRows;
+ if (filteredRows > 0) m.rowsExcludedWithinRowGroup().add(filteredRows);
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ RowRanges readerRanges = readerRangesFor(finalRanges, finalRowCount, blockRowCount);
+ installRowGroup(dataPages, readerRanges, keptRows);
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queues at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ return spliceCurrentRowGroup && isKeyTopLevel[slot];
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses, since the rows its column index already excluded were never this filter's
+ // to save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ *
+ * <p>A file this reader can no longer read in part still comes through here, since these are the
+ * rows a plain read of the same query would read, and leaving them to the post-scan Filter
+ * instead would make the scan slower than having no storage filter at all.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * What the surviving rows of the current row group cost to hold as row ranges, which is the half
+ * of the budget that a row group buffering nothing still pays. A filter whose survivors are
+ * scattered makes one range per surviving row, and phase 2 needs the whole set to select its
+ * pages. One set is held, not one per column reader, and the readers of a row group walk that
+ * same list, each with a cursor of its own.
+ */
+ private static long rowRangeStateBytes(long rangeCount) {
+ // Parquet's own `RowRanges.Range` holds two longs, plus its object header and a list slot.
+ return rangeCount * 40L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group. The
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ ensureKeyScratchAllocated();
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below, which is the bytes buffered
+ // for splicing and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so whether anything survives
+ // at all was the only thing left to learn, and it does. Stop here, since the ranges are
+ // not needed and the rest of the key column need not be decoded or evaluated. The
+ // row groups the filter empties are still skipped, which is the case that returns a
+ // non-null empty range set below.
+ abandonSplicing();
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) splicedBytes += appendSurvivorRowToAccumulators(r);
+ // Both halves of what this row group retains grow per survivor, and either can cross
+ // the budget on its own, so they are weighed together here and nowhere else. The cheaper
+ // concession comes first, which is to release the buffer, and the filter goes too if the
+ // ranges alone still do not fit.
+ long rangeBytes = rowRangeStateBytes(survivorRangeCount);
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper concession first, releasing the buffer, which is a no-op for a row group
+ // that has already given splicing up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ if (rangeBytes > cap) {
+ // The ranges being built are about to be thrown away, so stop evaluating the rest.
+ return null;
+ }
+ }
+ } catch (Exception e) {
+ // Fail open, whatever the exception is and wherever in the row it arose. The predicate
+ // ran on a row that, in the plan, an earlier conjunct would have rejected before it, so a
+ // plain scan never evaluates it there. Giving the filter up for this row group puts every
+ // row the pushed filter kept back in the output, and the post-scan Filter then evaluates
+ // the conjuncts in their own order. Either an earlier one drops the row before this
+ // expression runs, or it does not and the query fails the way it would have without this
+ // feature.
+ //
+ // Not narrowed to Spark's own errors, because a built-in expression can raise a plain JDK
+ // one, as `timestamp_seconds` of a decimal throws ArithmeticException from
+ // longValueExact. Nor to unchecked ones, because a Hive UDF in the key throws a checked
+ // SparkException (FAILED_EXECUTE_UDF) that nothing declares. Rethrowing has two bad ends,
+ // and neither is the error the plan would have raised. The query fails where a plain read
+ // succeeds, and under `ignoreCorruptFiles` this reader's exception is read as a corrupt
+ // file, which drops the rest of it silently.
+ //
+ // Copying a surviving key lands here too, which a survivor vector that cannot grow does,
+ // and reading the row group with nothing buffered is the answer to that as well.
+ logFilterGivenUpOnError(e);
+ abandonSplicing();
+ return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // Any partially-filled accumulator goes to its queue, so the emit path can dequeue it as this
+ // row group's final batch.
+ if (spliceCurrentRowGroup && currentKeyAccumulatorRowCount > 0) {
+ pushAccumulatorsToQueues();
+ }
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // The row the predicate evaluates holds the key columns phase 1 reads, and for a key column
+ // this file does not have, the constant vector `initBatch` built for it. `reset()` is a no-op
+ // on those, and this reader never closes them, since the emitted batch owns them.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors as the emitted batch holds them, in the order the predicate's row
+ * reads them. Which is what a missing key column reads as, and what phase 1 overwrites for a key
+ * column this file has. Taking a present key's batch vector and discarding it is a getter over a
+ * final field, and it saves holding the missing positions as state.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = columnVectors[keyIndices[p]].getValueVector();
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows
+ * each.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ currentKeyAccumulators[i] = newKeyVector(i);
+ }
+ }
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, handing them to their
+ * queues first if the last row filled them. All key columns advance in lockstep, which is what
+ * keeps the queues aligned. Returns the bytes the row added, which the caller weighs against its
+ * budget.
+ */
+ private long appendSurvivorRowToAccumulators(int srcRow) {
+ if (currentKeyAccumulatorRowCount == capacity) {
+ // Here rather than after the append that fills them, so that a row group whose survivors are
+ // a multiple of the capacity leaves no unused set of accumulators behind.
+ pushAccumulatorsToQueues();
+ ensureCurrentKeyAccumulatorsAllocated();
+ }
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final KeyColumn[] keys = keyColumns;
+ long valueBytes = 0L;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ KeyColumn key = keys[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ key.copier().copy(dst, dstRow, src, srcRow);
+ // Measured on the destination, because a dictionary-encoded source has no length of its
+ // own, its values being read through the dictionary.
+ if (key.variableLength()) {
+ valueBytes += VARIABLE_LENGTH_BYTES_FACTOR * dst.getArrayLength(dstRow);
+ }
+ }
+ }
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ return keyFixedBytesPerRow + valueBytes;
+ }
+
+ /**
+ * Hands every accumulator to its queue, which is what the emit path dequeues from. The row count
+ * goes with them, since it describes what the accumulators hold and they now hold nothing.
+ */
+ private void pushAccumulatorsToQueues() {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /** Releases the pages phase 2 read for the row group just emitted, if any. */
+ private void closeDataPages() {
+ if (dataPages != null) {
+ dataPages.close();
+ dataPages = null;
+ }
+ }
+
+ /**
+ * Reports the first row group of this file whose filter could not be evaluated. Once per file,
Review Comment:
Reworded everywhere it said file: the log javadoc and flag, `ParquetBlockAccounting`'s "one instance per reader, which reads one split", `partialReadUnavailable`, the retry's comment, and the description's "at most once per split" and "latches for the rest of the split".
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2755 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.SparkException
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = readerFn(file).asInstanceOf[Iterator[Object]].map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter =
+ keyTypeAtLeastFilter(Literal(threshold), LongType, metrics)
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ // Reads a key-type fixture, returning the survivor keys and the reader. The `extract` function
+ // pulls one value at a time from the batch's key column.
+ private def readKeysAll[T](
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T,
+ capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter, (batch, i) => extract(batch.column(0), i),
+ capacity)
+
+ // Builds a `k >= threshold` storage filter bound to position 0 against a `(k, v)` schema whose
+ // key has the given type.
+ private def keyTypeAtLeastFilter(
+ threshold: Literal,
+ keyType: DataType,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, keyType, nullable = false), threshold)
+ val requested = StructType(Seq(
+ StructField("k", keyType, nullable = false), StructField("v", StringType, nullable = true)))
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("mixed: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ // Output is exact. VectorizedColumnReader uses PageReadStore.getRowIndexes (driven by
+ // our finalRanges) to skip rows within partial pages, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted has
+ // to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(195L))
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs the reader's own initialization, and it
+ // used to run that a second time on top. A reader that declined the filter on the first pass,
+ // here because every projected column is a key column, then reached the filter it had just
+ // released and failed with an NPE instead of reading plainly.
+ withTempDir { dir =>
+ val keys = (1L to 20L)
+ val path = writeKeyParquetFile(dir, keys)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(15L))),
+ requested)
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ Utils.tryWithResource(new LateMaterializationParquetRecordReader(false, 4096, filter)) {
+ reader =>
+ reader.initialize(split, context)
+ reader.initBatch(new StructType(), null)
+ val read = mutable.ArrayBuffer[Long]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ assert(read == keys, s"every row must come back unfiltered; got ${read.size}")
+ }
+ }
+ }
+
+ test("multi-batch emit: survivor count exceeds capacity") {
+ // Drive the reader at capacity = 16 with a row group of 71 survivors out of 100. Exercises:
+ // - The per-key-column queue holding multiple full-capacity vectors plus a partial tail.
+ // - The published queue head getting closed at the start of every subsequent emit.
+ // - The batch's key slots being rewritten ceil(71/16) = 5 times.
+ // The filter keeps most of the row group rather than all of it, so a reader that declined the
+ // filter altogether would return more rows than this asserts rather than passing.
+ withTempDir { dir =>
+ val keys = (1L to 100L)
+ // Big rowGroupSize so all 100 rows fit in one row group.
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 64 * 1024L)
+ val filter = keyTypeAtLeastFilter(Literal(30L), LongType)
+ val (result, reader) =
+ readKeysAll(path, filter, (vec, i) => vec.getLong(i), capacity = 16)
+ try {
+ assert(result == keys.filter(_ >= 30L).toSeq,
+ s"expected the surviving keys in order across multiple batches; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("int key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the IntegerType branch of ValueCopier.
+ withTempDir { dir =>
+ val keys = (1 to 100)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyTypeAtLeastFilter(Literal(90), IntegerType)
+ val (result, reader) = readKeysAll(path, filter, (vec, i) => vec.getInt(i))
+ try {
+ assert(result.toSet == keys.filter(_ >= 90).toSet,
+ s"expected int keys >= 90; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("string key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the StringType branch of ValueCopier (variable-length byte copy via putByteArray).
+ withTempDir { dir =>
+ val keys = (1 to 20).map(i => f"k$i%03d")
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyTypeAtLeastFilter(
+ Literal.create("k015", StringType), StringType)
+ val (result, reader) =
+ readKeysAll(path, filter, (vec, i) => vec.getUTF8String(i).toString)
+ try {
+ assert(result.toSet == keys.filter(_ >= "k015").toSet,
+ s"expected string keys >= 'k015'; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(0L))),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ Literal(bloomBytes(BloomFilter.create(10))),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // Serializes a [[BloomFilter]] to bytes suitable for a [[Literal]].
+ // The same serialization a runtime bloom's Literal carries, so these tests build theirs the way
+ // the planner does.
+ private def bloomBytes(bf: BloomFilter): Array[Byte] = BloomFilterAggregate.serialize(bf)
+
+ // Computes `XxHash64(v)` using the same seed Spark uses for runtime bloom filters.
+ private def xxHash64(v: Any, dt: DataType): Long = {
+ new XxHash64(Seq(Literal(v, dt))).eval(InternalRow.empty).asInstanceOf[Long]
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = StructType(Seq(
+ StructField("k", decimalType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = true)))
+ val key = ThrowsCheckedOn(BoundReference(0, LongType, nullable = false), bad = 7L)
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // Returns the FileSourceScanExec for a parquet read, with `storageFilters` attached. Goes through
+ // Spark's normal planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind, the SQL conf check,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def scanWithStorageFilter(
+ path: String,
+ keyName: String,
+ threshold: Long): FileSourceScanExec = {
+ val df = spark.read.parquet(path).select("k", "v")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ val keyAttr = scan.output.find(_.name == keyName).getOrElse(fail(s"No $keyName in scan output"))
+ val expr = GreaterThanOrEqual(keyAttr, Literal(threshold))
+ scan.copy(storageFilters = Seq(expr))
+ }
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect().map(r => (r.getLong(0), r.getString(1)))
+ }
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 195L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sortBy(_._1)}; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. Phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ assert(scan.simpleString(200).contains("GreaterThan(v,"),
+ s"the data filter must actually be pushed for this test to mean anything: " +
+ scan.simpleString(200))
+ // Storage filter on `k` (key column).
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(
+ storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(100L))))
+
+ val collected = executePlanCollect(withSF).toSet
+ // Every row satisfies v > 'v_000', so the storage filter alone decides the result.
+ val expected = rows.filter(_._1 >= 100L).toSet
+ assert(collected == expected,
+ s"got ${collected.toSeq.sortBy(_._1)}; expected ${expected.toSeq.sortBy(_._1)}")
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms >= 1,
+ s"expected >= 1 bloom filter on scan.storageFilters; got $storageBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSet == Set(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves bloom filter as post-scan FilterExec when conf is off") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters when conf is off; got $storageBlooms")
+ assert(postScanBlooms >= 1,
+ s"expected bloom in post-scan FilterExec when conf is off; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("the feature still engages when ignoreCorruptFiles is on") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSet == Set((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since the
+ // post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty).getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+
+ test("a row group over the splice cap is read the plain way, same rows, more bytes") {
+ // Past the cap a row group is read the plain way. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Two things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. And
+ // the fallback must actually have happened, which is observable. It reads the key column a
+ // second time, so it transfers strictly more bytes. Without that second assertion the test
+ // would pass even if the cap never reached the reader.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group read the plain way past the cap still
+ // has to emit in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") {
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, fileSchema, new StructType(), fileSchema, Nil, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), countingHadoopConf(), metricMap())
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+ readCounting(path, readerFn)
+ }
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the
+ // row ranges. `k >= 350` keeps a contiguous run, so there is one range to hold.
+ val (plainRows, plainBytes) = run("100b")
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true))),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity = 16)
+ try {
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected a row group with no survivor at all; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group past the cap charges its second key read against the byte metric") {
+ // Giving splicing up means phase 2 reads the key columns a second time, while the baseline
+ // counts them once, in phase 1. That extra read is a cost against the saving, so the same file
+ // and filter must report less avoided than they do while splicing. Without that term the metric
+ // would credit the feature with bytes it did transfer.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+
+ def avoidedBytes(cap: Long, threshold: Long): Long = {
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))),
+ fileSchema, m, cap)
+ val (_, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => b.column(0).getLong(i), capacity = 16)
+ try m.bytesAvoidedByPageFiltering.value finally reader.close()
+ }
+
+ val spliced = avoidedBytes(64L * 1024 * 1024, 350L)
+ val plain = avoidedBytes(100L, 350L)
+ assert(spliced > 0, s"page filtering has to avoid something here; got $spliced")
+ assert(plain < spliced,
+ s"the second key read must count against the saving; plain=$plain spliced=$spliced")
+
+ // The two halves of the budget are weighed together, and this is the case that says so. Eight
+ // survivors of a long key buffer 8 * (1 + 8) = 72 bytes and fall in one range of 40, so
+ // neither half reaches the 100-byte cap on its own while the sum passes it at the seventh.
+ // Weighing them separately would keep splicing here, and report the larger saving for it.
+ val eightSpliced = avoidedBytes(64L * 1024 * 1024, 393L)
+ val eightPlain = avoidedBytes(100L, 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+ }
+ }
+
+ // Writes a parquet file the low-level way, with no offset index for any column.
+ private def writeParquetFileWithoutOffsetIndex(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]]): String =
+ writeParquetFileByHand(dir, blocks, (_, _) => false)
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The three row groups are ordered so that
+ // the file's first partial read is what discovers the missing index, and the two row groups
+ // after that discovery still get phase 1. The middle one is emptied by the filter and skipped
+ // whole, the last keeps rows and is read whole. The missing index is learned from the read that
+ // fails, so the first row group is the one that discovers it. Spark's own writer always writes
+ // the page index, hence the hand-built file.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileWithoutOffsetIndex(dir, blocks)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", LongType, nullable = false)))
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ // `withLogAppender` returns Unit, so the read's result is captured out of the block.
+ val logAppender = new LogAppender("no offset index")
+ var read: (Seq[(Long, Long)], VectorizedParquetRecordReader) = null
+ withLogAppender(logAppender) {
+ read = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ val (result, reader) = read
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering")),
+ "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index does not cost the file its page filtering") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, and the
+ // reader answers that by not trusting the page index for the rest of the file. Every row group
+ // after it would then be read whole, which is more than a plain read of the same query, since
+ // that keeps the pushed data filter's page pruning.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", LongType, nullable = false)))
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(50L)))
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"),
+ spark.sessionState.newHadoopConf(), metricMap())
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ val logAppender = new LogAppender("no offset index")
+ var values: Seq[Long] = null
+ withLogAppender(logAppender) {
+ values = readerFn(file).asInstanceOf[Iterator[Object]].flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(1).getLong)
+ case row: InternalRow => Seq(row.getLong(1))
+ }.toSeq
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering")),
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning, which is the regression this pins. Read whole they
+ // would contribute 200 rows rather than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(ref.getOffset)
+ raf.write(Array.fill(ref.getLength)(0xFF.toByte))
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry leaves the reader unable to narrow the rest of this file, since the next row
+ // group's store may be as unreadable as this one's. What it must not give up is the pushed data
+ // filter's own page pruning, which parquet did before the reader saw the block, and which a
+ // plain read of the same query keeps. Dropping that would read more than having no storage
+ // filter at all.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", LongType, nullable = false)))
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] = {
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"),
+ spark.sessionState.newHadoopConf(), metricMap())
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ readerFn(file).asInstanceOf[Iterator[Object]].flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(0).getLong)
+ case row: InternalRow => Seq(row.getLong(0))
+ }.toSeq
+ }
+ val logAppender = new LogAppender("damaged offset index")
+ var filtered: Seq[Long] = null
+ withLogAppender(logAppender) {
+ filtered = read(evenKeys)
+ }
+ assert(logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering")),
+ "the read must have discovered the damaged index, or this test proves nothing")
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry has latched.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the splice queues and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString), capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // The capacity decides where the budget is crossed. At 4096 the 200 survivors never fill an
+ // accumulator, so the ranges alone cross it; at 16 the buffer has been through several
+ // accumulators by then. Both have to abandon the buffer and give the filter up together.
+ // Keeping one without the other would leave phase 2 reading non-key columns while emit
+ // spliced keys from a queue holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true))),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString))
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSet == Set("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(42L, LongType))
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does. The reader never calls `initialize(partitionIndex)`, so it could not evaluate
+ // the expression, while the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for. A dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`, since `/` is floating-point division in Spark SQL, which would hand every
+ // key expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize, dictionary = dictionary)
+ }
+
+ // Every column chunk of the file, from its footer. The facts the tests below assert about a
+ // fixture are read off these rather than assumed from the write options.
+ private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] =
+ footerOf(filePath).getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq
+
+ private def footerOf(filePath: String): ParquetMetadata = {
+ val conf = spark.sessionState.newHadoopConf()
+ ParquetFooterReader.readFooter(
+ HadoopInputFile.fromPath(new Path(filePath), conf), ParquetMetadataConverter.NO_FILTER)
+ }
+
+ // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String, column: String = "k"): Set[Encoding] =
+ footerChunks(filePath).filter(_.getPath.toDotString == column)
+ .flatMap(_.getEncodings.asScala).toSet
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // builds the plain vectorized reader rather than the late-materialization one. Every read helper
+ // in this suite goes through here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need. The caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ if (storageFilter == null) {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } else {
+ new LateMaterializationParquetRecordReader(useOffHeap, capacity, storageFilter)
+ }
+ } { reader =>
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // A key value as a string in its *internal* representation, so the splicing path and the plain
+ // path can be compared without knowing the type and without external type conversion.
+ // `InternalRow.getAccessor` does the per-type read, which is why this takes the batch's row
+ // rather than the column vector.
+ private def renderValue(row: InternalRow, ordinal: Int, dt: DataType): String =
+ InternalRow.getAccessor(dt)(row, ordinal) match {
+ case null => "null"
+ // Two equal arrays are two objects, so the default toString would compare identities.
+ case bytes: Array[Byte] => bytes.mkString(",")
+ case value => value.toString
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path. Whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.getRow(i), 0, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage, one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why. There is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ } {
+ val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded"
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = StructType(Seq(
+ StructField("k", dt, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // This is a regression test. TimeType is an AtomicType that passes every planning-time gate
+ // (it is batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a
+ // TIME join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = StructType(Seq(
+ StructField("k", dt, nullable = true), StructField("v", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // What FileSourceStrategy offers, what ParquetStorageFilter.create admits and what the reader
+ // can copy are one list, because isSupportedKeyType answers from ValueCopier. Both are asserted
+ // here. A type the copier has no case for would be a task failure rather than a planning-time
+ // rejection, and the delegation is what rules that out.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ assert(ValueCopier.forType(dt) != null, s"$dt should have a value copier")
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ intercept[IllegalStateException](ValueCopier.forType(dt))
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = true), StructField("v", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("two key columns: both accumulators stay aligned with each other and with the data column") {
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300 => a in [100, 150]
+ val bound = And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)),
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1), "both key columns should be recognized")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, and the emit path depends on
+ // that. It pairs the k-th key slot of the batch with key-row position k, which is the identity
+ // only while the list is ascending. Nothing else in the suite supplies an unsorted one, because
+ // every other fixture happens to mention its keys in column order. Production does not promise
+ // that. The conjuncts arrive in `afterScanFilters` order, which follows the join. Without the
+ // sort this returns `a` and `b` swapped, with no exception to notice.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // Same predicate as the test above, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // The emit path rewrites the key slots alone, and a partition slot is not one of them. It sits
+ // past the data columns and must keep the constant vector `initBatch` populated for it.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite. OffHeapColumnVector.close() frees
+ // the native buffer, so a read after close is a crash rather than stale data. A second close
+ // is harmless, since close() zeroes the addresses it freed. capacity = 16 over 71 survivors
+ // forces 5 emits, each closing the previous emit's dequeued key vectors. The other 29 rows
+ // are what makes a declining reader fail this.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(30L))
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (30L to 100L),
+ s"expected the 71 surviving keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test(s"$mode vectors: row-group skipping and page filtering over a mixed projection") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(195L, m)
+ val (collected, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString),
+ useOffHeap = useOffHeap)
+ try {
+ assert(collected == rows.filter(_._1 >= 195L),
+ s"expected exact filtering; got ${collected.map(_._1)}")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and the
+ // partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // One ColumnarBatch is handed out for the whole read and its key slots are rewritten per
+ // batch, so a consumer holding on to an earlier getCurrentValue() would read the wrong
+ // vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new LateMaterializationParquetRecordReader(false, 8, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution, a column missing from the physical file -----
+
+ test("non-key column missing from a file: byte-avoided metrics tolerate a missing offset index") {
+ // ColumnIndexStore returns a null OffsetIndex for a column that is in the (clipped) requested
+ // schema but absent from the row group, which is exactly what schema evolution produces. The
+ // avoided-bytes walk runs over the full requested schema on every row group of every file, so
+ // without a null guard this is an NPE on the first row group of the older file.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // The older file has (k, v), the newer one (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect()
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSet
+ val expected = ((150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))).toSet
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // The avoided-bytes counters are what walk the offset index of the missing column, so a
+ // positive total is the evidence that the walk ran and coped. Asserting non-negativity
+ // would prove nothing. `SQLMetric.add` drops a negative and the value cannot go below zero.
+ val bytesRg = withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value + bytesPf.value > 0,
+ s"the walk must credit some avoided bytes; got rg=${bytesRg.value} pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = scanOf(spark.read.parquet(path).select("k", "v").queryExecution.executedPlan)
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Set[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Set[(Long, Option[Long])]) = {
+ val df = spark.sql(s"SELECT id, k FROM $table")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(buildFilter(keyAttr)))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = scan.output.indexWhere(_.name == "id")
+ val kPos = scan.output.indexWhere(_.name == "k")
+ val collected = rowPlan.executeCollect()
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSet
+ (withSF, collected)
+ }
+
+ test("all keys missing: an error on the constant reads the file rather than skipping it") {
+ // A file with none of the filter's key columns makes the predicate constant over it, so the
+ // reader decides once, from the values `initBatch` materialized for those columns. That
+ // evaluation can throw exactly as a row's value can. Here the older file's `k` reads as its
+ // DEFAULT, which is not a number, and the cast fails under ANSI. Failing open reads the file
+ // the way a plain scan would, so its rows survive. Skipping it would lose them, and rethrowing
+ // would fail a query a plain read answers.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true",
+ SQLConf.ANSI_ENABLED.key -> "true") {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql("ALTER TABLE evolved ADD COLUMN k STRING DEFAULT 'not-a-number'")
+ spark.sql("INSERT INTO evolved VALUES (4, '40'), (5, '50')")
+ val scan = scanOf(spark.sql("SELECT id, k FROM evolved").queryExecution.executedPlan)
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters =
+ Seq(GreaterThanOrEqual(Cast(keyAttr, LongType), Literal(1L))))
+ val idPos = scan.output.indexWhere(_.name == "id")
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val ids = rowPlan.executeCollect().map(_.getLong(idPos)).toSet
+ assert(ids == Set(1L, 2L, 3L, 4L, 5L),
+ s"the older file must fail open and the newer one pass the cast; got $ids")
+ }
+ }
+ }
+
+ test("all keys missing: a checked exception on the constant reads the file") {
+ // The constant evaluation fails open under the same rule as a row's, a checked exception
+ // included. The older file's `k` reads as its DEFAULT 7, which the key expression throws on,
+ // so that file must be read the way a plain scan would. The newer file's keys evaluate
+ // normally and are rejected, since no Filter runs above this scan.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(ThrowsCheckedOn(k, bad = 7L), Literal(100L)))
+ assert(collected == Set((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L))),
+ s"the older file must fail open and the newer one be filtered; got $collected")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test. The default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Set[(Long, Option[Long])] = Set((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(StorageFilterMetrics.ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Set((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got ${rejectsNull.toSeq.sorted}")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got ${acceptsNull.toSeq.sorted}")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect().map(r => (r.getLong(0), r.getLong(2))).toSet
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1)).toSet
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got " +
+ s"${collected.toSeq.sorted.take(5)}")
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select("k", "s")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(195L))))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect()
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSet
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k")).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sorted}; expected $expected")
+ }
+ }
+ }
+
+ test("a storage filter binds to the right column with partition and metadata columns in output") {
+ // `preparedStorageFilters` binds the filter's references against `output.take(requiredSchema
+ // .length)`, and `output` is the data columns, then the generated metadata columns, then the
+ // partition columns, then the constant metadata ones. A projection carrying all four is where a
+ // wrong prefix would bind the predicate to another column and drop rows the filter keeps, which
+ // the post-scan Filter cannot put back.
+ withTempDir { dir =>
+ val outDir = new File(dir, "parted").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "CONCAT('v_', id) AS v", "CAST(id % 3 AS INT) AS p")
+ .repartition(1)
+ .write
+ .partitionBy("p")
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select(
+ col("k"), col("v"), col("p"),
+ col("_metadata.row_index").as("ri"), col("_metadata.file_size").as("fs"))
+ val scan = scanOf(df.queryExecution.executedPlan)
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+ // The assumption the binding rests on, stated where a change to `output` would break it.
+ assert(withSF.output.take(withSF.requiredSchema.length).map(_.name) ==
+ withSF.requiredSchema.fieldNames.toSeq,
+ s"the scan's leading attributes must be its data columns; got " +
+ s"${withSF.output.map(_.name)}")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val pPos = withSF.output.indexWhere(_.name == "p")
+ def collectAll(plan: SparkPlan): Set[Seq[Any]] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ val types = plan.output.map(_.dataType)
+ rowPlan.executeCollect().map(_.toSeq(types)).toSet
+ }
+ val filtered = collectAll(withSF)
+ // The same scan without the filter is the oracle, so the row indexes and file sizes come
+ // from the read rather than from arithmetic over how the partitions were written.
+ val expected = collectAll(withSF.copy(storageFilters = Nil))
+ .filter(_(kPos).asInstanceOf[Long] >= 150L)
+ assert(filtered == expected,
+ s"expected ${expected.size} rows with their own partition and metadata values; " +
+ s"got ${filtered.size}")
+ assert(filtered.map(_(pPos)).size == 3,
+ s"every partition value must still be represented; got ${filtered.map(_(pPos))}")
+ }
+ }
+ }
+
+ // ----- Planner gates and the lost-filter invariant -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false. It may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ //
+ // This hook answers about shapes and types alone. Whether the conjunct is deterministic and
+ // whether its references are projected columns of this scan are plan-visible, so they stay with
+ // the planner; `FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter`
+ // covers that half.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ // A cast key is supported. In ANSI mode it can throw on a row an earlier conjunct would have
+ // dropped, and the reader handles that where it arises, by giving the filter up for the row
+ // group. Declining here instead would also decline every widening cast, which is what type
+ // coercion inserts for a join between an int and a bigint column.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+
+ // The conf is read in the format rather than in the planner, and it is the per-scan question
+ // that carries it, asked once before anything per conjunct.
+ assert(!format.supportsStorageFilterPushdown(spark), "the feature is off with the conf off")
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ assert(format.supportsStorageFilterPushdown(spark), "and on with it on")
+ assert(!subclass.supportsStorageFilterPushdown(spark), "a subclass must not claim support")
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ assert(format.supportsStorageFilter(onCast),
+ "a cast key is pushed, and an evaluation error gives the row group up")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The whole lost-filter safety argument rests on this gate. If the reader cannot do late
+ // materialization, the planner must NOT move the bloom out of the post-scan Filter.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one. Every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, "k", threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSet
+ assert(keys == (1L to 50L).toSet,
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support declines rather than delegating") {
+ // The default `FileFormat.buildReaderWithStorageFilters` answers None, and the caller falls
+ // back to the ordinary builder. That is what keeps the two builders from being able to call
+ // each other, and declining is safe because the post-scan Filter still holds the conjunct.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(1L)))
+ val declined = new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration(), Map.empty)
+ assert(declined.isEmpty, "the default must not build a reader of its own")
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilterPushdown(spark),
+ "and it must not claim support either")
+ }
+
+ test("a scan whose format declines falls back to the ordinary reader and returns every row") {
+ // The check above is that the default answers None. This is the other half, that `inputRDD`
+ // then builds the ordinary reader and the scan still produces its rows. A `ParquetFileFormat`
+ // subclass is the shape that reaches it, since it inherits the override and declines by class,
+ // and its ordinary builder is the one an extension would have customized.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanOf(spark.read.parquet(path).select("k", "v").queryExecution.executedPlan)
+ val keyAttr = scan.output.find(_.name == "k").get
+ val declining = scan.copy(
+ relation = scan.relation.copy(fileFormat = new ParquetFileFormat() {})(spark),
+ storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(195L))))
+ val collected = executePlanCollect(declining).toSet
+ assert(collected == rows.toSet,
+ s"a declining format reads the file plainly; got ${collected.size} of ${rows.size} rows")
+ }
+ }
+ }
+
+ Seq(false, true).foreach { aqe =>
+ test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") {
+ // AQE is on by default in production, and it is where the bloom subquery is planned by
+ // PlanAdaptiveSubqueries rather than PlanSubqueries, which is the path
+ // preparedStorageFilters' ScalarSubquery materialization depends on.
+ withBloomFilterTables {
+ val baseConf = Map(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString)
+ def run(pushdown: Boolean): (Int, Set[(Long, Long)]) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, rows) = runBloomFilterJoin()
+ (countBloomFiltersInStorageFilters(plan),
+ rows.map(r => (r.getLong(0), r.getLong(1))).toSet)
+ }
+ val (offFilters, off) = run(false)
+ val (onFilters, on) = run(true)
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ // Equal results are guaranteed by the post-scan Filter whether or not the scan applied the
+ // filter, so they alone would not notice AQE dropping it on the way to the final plan.
+ assert(offFilters == 0 && onFilters == 1,
+ s"the scan must carry the filter with the conf on and not with it off; " +
+ s"got on=$onFilters off=$offFilters")
+ }
+ }
+ }
+
+ test("canonicalization keeps a storage-filter scan distinct from a plain one") {
+ // storageFilters is in doCanonicalize and in the case class equality, which is what stops
+ // exchange and subquery reuse from serving one scan's result to the other. Reuse compares
+ // canonicalized plans, so a scan that filters must not match a plain scan of the same file, and
+ // two scans carrying the same filter must still match.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, "k", threshold = 25L)
+ val plain = withSF.copy(storageFilters = Nil)
+ assert(withSF != plain, "case class equality must take storageFilters into account")
+ assert(!withSF.sameResult(plain),
+ s"a filtering scan must not be reusable as a plain one:\n${withSF.canonicalized}\n" +
+ s"${plain.canonicalized}")
+ // A second, independently planned scan of the same file with the same filter still
+ // matches, so reuse is not disabled wholesale. Its key attribute carries a different
+ // exprId, which is what canonicalization normalizes away.
+ val sameSF = scanWithStorageFilter(path, "k", threshold = 25L)
+ assert(withSF.sameResult(sameSF),
+ s"two scans with the same storage filter must stay reusable:\n" +
+ s"${withSF.canonicalized}\n${sameSF.canonicalized}")
+ val otherSF = scanWithStorageFilter(path, "k", threshold = 30L)
+ assert(!withSF.sameResult(otherSF), "a different threshold is a different result")
+ }
+ }
+ }
+
+ test("whole-stage codegen off: the row-at-a-time path still applies the extracted bloom") {
+ // The planner's extraction gate is `fileFormat.supportBatch`, which does not look at
+ // whole-stage codegen, while `FileSourceScanExec.supportsColumnar` does. So with codegen off
+ // the bloom is still extracted, `returningBatch` is false, and the reader serves the spliced
+ // batch one row at a time. That is the only combination where the planner's gate is weaker
+ // than the runtime's.
+ withBloomFilterTables {
+ val baseConf = Map(
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "false")
+ def run(pushdown: Boolean): (Set[(Long, Long)], Int, Boolean) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, result) = runBloomFilterJoin()
+ val columnar = plan.collect { case s: FileSourceScanExec => s.supportsColumnar }
+ (result.map(r => (r.getLong(0), r.getLong(1))).toSet,
+ countBloomFiltersInStorageFilters(plan), columnar.forall(_ == false))
+ }
+ val (off, _, _) = run(false)
+ val (on, storageBlooms, noColumnarScan) = run(true)
+ assert(storageBlooms >= 1, s"the bloom must still be extracted with codegen off; " +
+ s"got $storageBlooms")
+ assert(noColumnarScan, "with codegen off no scan should output columnar batches")
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ }
+ }
+
+ test("off-heap column vectors through the planner: spliced values survive the free") {
+ // Off-heap is where the vector lifecycle actually bites. The previous batch's key vectors are
+ // freed at the next nextBatch(), so a stale reference reads released native memory rather than
+ // old bytes. This drives it through the planner, where the batch also crosses
+ // ColumnarToRowExec.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.COLUMN_VECTOR_OFFHEAP_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 150L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 150L).toSet
+ assert(collected == expected,
+ s"got ${collected.size} rows; expected ${expected.size}. " +
+ s"first few: ${collected.toSeq.sortBy(_._1).take(3)}")
+ }
+ }
+ }
+
+ // ----- Page-level pushedFilterRanges (a strict subset of the row group) -----
+
+ test("pushed data filter narrows to a page subset: phase 1 stays aligned with the row indexes") {
+ // Every other fixture writes one page per column per row group, so column-index filtering can
+ // only ever drop whole row groups and `pushedFilterRanges` is always the entire block. That
+ // makes the phase 1 alignment hold trivially, that the r-th row readBatch delivers pairs with
+ // rowIndexIter.nextLong(). With a small page size the data filter narrows
+ // to a page subset, so the two sequences only agree if the pairing is actually correct.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // `v` is correlated with `k`, so a range predicate on v prunes pages, not whole row groups.
+ val df = spark.read.parquet(path).select("k", "v").filter("v >= 'v_0300'")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ assert(scan.simpleString(200).contains("GreaterThanOrEqual(v,"),
+ s"the data filter must be pushed: ${scan.simpleString(200)}")
+ val keyAttr = scan.output.find(_.name == "k").get
+ // Storage filter keeps a band that starts inside the data filter's surviving range.
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(350L))))
+
+ val collected = executePlanCollect(withSF).toSet
+ val expected = rows.filter(r => r._2 >= "v_0300" && r._1 >= 350L).toSet
+ assert(collected == expected,
+ s"got ${collected.size} rows; expected ${expected.size}. " +
+ s"first few: ${collected.toSeq.sortBy(_._1).take(3)}")
+ }
+ }
+ }
+
+ test("page-subset ranges still credit the avoided bytes") {
+ // The strict-subset branch of compressedBytesForRowRanges (offset index + dictionary page) only
+ // runs when pushedFilterRanges is narrower than the block, which needs a multi-page row group.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v >= 'v_0100'")
+ val scan = scanOf(df.queryExecution.executedPlan)
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(390L))))
+ val collected = executePlanCollect(withSF).toSet
+ assert(collected == rows.filter(_._1 >= 390L).toSet, s"got ${collected.size} rows")
+
+ val bytesRg = withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING)
+ // Non-negativity is not assertable. `SQLMetric.add` drops a negative, so the value cannot
+ // go below zero however wrong the arithmetic is. A positive total can fail.
+ assert(bytesRg.value + bytesPf.value > 0,
+ s"page-subset ranges must still credit avoided bytes; rg=${bytesRg.value} " +
+ s"pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ test("column-index filtering off: the filter keeps only the row groups it empties") {
+ // `parquet.filter.columnindex.enabled=false` is the escape hatch for a file whose page index is
+ // wrong, and it has to cover phase 2 as well as phase 0. Phase 2 reads part of a row group
+ // through the offset index, which parquet consults whatever that conf says, so a wrong index
+ // there would pair a row's key with another row's values. The post-scan Filter cannot catch
+ // that, since the key it sees is the right one.
+ //
+ // What the filter keeps in that case is the row groups it empties, which needs no index at all.
+ // So the two arms differ in granularity, not in correctness. With the conf on the scan returns
+ // exactly the surviving rows, with it off the row groups that hold one come back whole. This
+ // test executes the scan alone, so nothing re-applies the conjunct above it, and no data filter
+ // is pushed, so statistics-level row-group pruning stays out of it.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 1024L, pageSize = Some(512L))
+
+ def run(columnIndex: Boolean): (Set[(Long, String)], Long, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED -> columnIndex.toString) {
+ val scan = scanOf(spark.read.parquet(path).select("k", "v").queryExecution.executedPlan)
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(350L))))
+ val collected = executePlanCollect(withSF).toSet
+ (collected,
+ withSF.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED).value,
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value)
+ }
+
+ val expected = rows.filter(_._1 >= 350L).toSet
+ val (rowsOff, skippedOff, withinOff) = run(columnIndex = false)
+ val (rowsOn, skippedOn, withinOn) = run(columnIndex = true)
+ assert(rowsOn == expected,
+ s"with the column index on, got ${rowsOn.size} rows; expected ${expected.size}")
+ assert(withinOn > 0 && skippedOn > 0,
+ s"the on arm must both skip row groups and filter inside one, or this test proves " +
+ s"nothing; skipped=$skippedOn within=$withinOn")
+ assert(expected.subsetOf(rowsOff) && rowsOff.size > expected.size,
+ s"with the column index off the surviving rows' row groups come back whole; got " +
+ s"${rowsOff.size} rows for ${expected.size} survivors")
+ assert(rowsOff.size < rows.size && skippedOff > 0,
+ s"and the row groups the filter empties are still skipped; got ${rowsOff.size} rows of " +
+ s"${rows.size}, skipped=$skippedOff")
+ assert(withinOff == 0,
+ s"nothing can be excluded inside a row group without the page index; got $withinOff")
+ }
+ }
+
+ Seq(true, false).foreach { pushDataFilter =>
+ test("the byte walks cost no IO of their own " +
+ s"(pushed data filter narrowing the row group = $pushDataFilter)") {
+ // The byte walks report IO that did not happen, so they must not cause any IO themselves,
+ // which would make the counters change what they measure. A storage filter that keeps every
+ // row reads exactly the pages a plain read of the same projection reads. Phase 1 reads the
+ // key pages and phase 2 the rest. So the two arms must transfer the same bytes over a
+ // filesystem that counts them, and a walk that read an offset index of its own would show up
+ // as a difference.
+ //
+ // Both range shapes are covered, because the walk answers them from different places. With a
+ // pushed data filter the column index narrows `pushedFilterRanges` to a page subset and the
+ // walk reads the offset index, which is free only because column-index filtering built and
+ // memoized the store first. Without one the range is the whole block and the answer comes
+ // from the footer's `getTotalSize()`, which is the case where nothing else has built that
+ // store.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ // Keeps every row, so the filter changes which pages are read in neither arm.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(0L)))
+ val pushedFilters =
+ if (pushDataFilter) Seq(sources.GreaterThan("v", "v_0100")) else Nil
+ val metrics = metricMap()
+
+ def run(withStorageFilter: Boolean): (Int, Long) = {
+ val hadoopConf = countingHadoopConf()
+ val options = Map(FileFormat.OPTION_RETURNING_BATCH -> "true")
+ val format = new ParquetFileFormat()
+ val readerFn = if (withStorageFilter) {
+ format.buildReaderWithStorageFilters(spark, schema, new StructType(), schema,
+ pushedFilters, storageFilters, options, hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+ } else {
+ format.buildReaderWithPartitionValues(spark, schema, new StructType(), schema,
+ pushedFilters, options, hadoopConf)
+ }
+ readCounting(path, readerFn)
+ }
+
+ val (emittedPlain, bytesPlain) = run(withStorageFilter = false)
+ val (emittedFiltered, bytesFiltered) = run(withStorageFilter = true)
+
+ assert(emittedFiltered == emittedPlain,
+ s"a filter that keeps every row must not change the rows; got " +
+ s"filtered=$emittedFiltered plain=$emittedPlain")
+ assert(bytesPlain > 0, "the counting filesystem must have seen the read at all")
+ assert(bytesFiltered == bytesPlain,
+ s"applying the filter must read the same bytes as a plain read of the same projection; " +
+ s"filtered $bytesFiltered, plain $bytesPlain")
+
+ // The walks really ran, and on the shape each arm is meant to exercise.
+ val excludedWithin =
+ metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 0,
+ s"the filter keeps every row, so nothing is excluded; got $excludedWithin")
+ if (pushDataFilter) {
+ assert(emittedFiltered < rows.size,
+ s"the pushed filter must narrow the ranges below the block, so the offset-index " +
+ s"branch is the one measured; emitted $emittedFiltered of ${rows.size}")
+ } else {
+ assert(emittedFiltered == rows.size,
+ s"with no pushed filter every row reaches phase 1, so the footer branch is the one " +
+ s"measured; emitted $emittedFiltered of ${rows.size}")
+ }
+ }
+ }
+ }
+
+ // ----- Metric arithmetic -----
+
+ test("row metrics account for every row of the file") {
+ // Ties the three count metrics to the file. Whatever is not emitted must have been avoided
+ // either by a whole-row-group skip or by page filtering. A sign flip or a mis-scoped schema in
+ // the accounting shows up here, which a `>= 0` assertion cannot catch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 150L)
+ val emitted = executePlanCollect(scan).length
+ val avoidedRg = scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP)
+ val avoidedPf =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP)
+ assert(emitted == 51, s"expected keys 150..200; got $emitted")
+ assert(emitted + avoidedRg.value + avoidedPf.value == rows.size,
+ s"emitted ($emitted) + avoided by row group (${avoidedRg.value}) + avoided by page " +
+ s"filtering (${avoidedPf.value}) should equal ${rows.size}")
+ }
+ }
+ }
+
+ // ----- Projection order and batch boundaries -----
+
+ test("non-key column before the key column: emit maps queues to the right batch slots") {
+ // The emit loop walks batch slots in ascending order and pulls survivor queues in order, so it
+ // relies on keyColumnIndices being sorted. Every other test puts the keys in the leading slots,
+ // where an off-by-one in that pairing is invisible.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(
+ dir,
+ spark.range(1, 201).selectExpr("CONCAT('v_', id) AS v", "id AS k", "id * 10 AS w"),
+ rowGroupSize = 256L)
+
+ // Key is `k`, at slot 1 of the (v, k, w) projection.
+ val bound = GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(195L))
+ val requested = StructType(Seq(
+ StructField("v", StringType, nullable = true),
+ StructField("k", LongType, nullable = true),
+ StructField("w", LongType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(1), "the key must be recognized at slot 1")
+
+ val (result, reader) = readAllWith(path, Seq("v", "k", "w"), filter,
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getLong(i),
+ b.column(2).getLong(i)))
+ try {
+ val expected = (195L to 200L).map(k => (s"v_$k", k, k * 10))
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("survivor count is an exact multiple of capacity: no partial trailing accumulator") {
+ // Phase 1's end-of-row-group push has nothing to do only when the last accumulator came out
+ // exactly full. 64 survivors at capacity 16 does that; the multi-batch tests use 71, which does
+ // not. The
+ // 64 are the tail of a 100-row row group, so a reader that declined the filter would return the
+ // other 36 as well rather than passing.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(37L))
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = true), StructField("v", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i), capacity = 16)
+ try {
+ assert(result == (37L to 100L), s"expected the 64 surviving keys; got ${result.size}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("early termination: the reader stops without draining the file") {
+ // executeTake on a bare ColumnarToRowExec goes through ColumnarToRowEvaluatorFactory, not
+ // through the generated code, so nothing closes the batch from outside here. What this covers
+ // is abandoning the reader mid-file. The survivor queues still hold vectors, and
+ // RecordReaderIterator closes the reader on task completion. The external close is the test
+ // below.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 50L)
+ val limited = ColumnarToRowExec(scan).executeTake(5)
+ assert(limited.length == 5, s"expected 5 rows from the limit; got ${limited.length}")
+ assert(limited.forall(_.getLong(0) >= 50L),
+ s"every row must satisfy the storage filter; got ${limited.map(_.getLong(0)).toSeq}")
+ }
+ }
+ }
+
+ test("a limit under whole-stage codegen closes the spliced batch from outside") {
+ // `batch.close()` is emitted by ColumnarToRowExec.doProduce alone, so it only runs under
+ // WholeStageCodegenExec, and only when the row loop exits with a batch still in hand. That exit
+ // is the limit check, which needs a limit inside the same codegen stage, so the plan is built
+ // with LocalLimitExec and handed to CollapseCodegenStages. The generated source is asserted to
+ // hold that close, which says no more than that `ColumnarToRowExec` is in the stage, since it
+ // emits that line unconditionally. What exercises the close is the collect below, where the
+ // limit makes the row loop exit with a batch still in hand.
+ //
+ // What it exercises is that the spliced batch's columns are closed from outside while the
+ // reader is still open, and the reader's own close() then runs over the same vectors.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 50L)
+ val planned =
+ CollapseCodegenStages().apply(LocalLimitExec(5, ColumnarToRowExec(scan)))
+ val stage = planned match {
+ case w: WholeStageCodegenExec => w
+ case other => fail(s"expected a whole-stage codegen plan, got $other")
+ }
+ val source = stage.doCodeGen()._2.body
+ assert(source.contains(".close();"),
+ s"the generated code must close the batch, which is this plan's whole point; " +
+ s"source:\n$source")
+
+ val limited = stage.executeCollect()
+ assert(limited.length == 5, s"expected 5 rows from the limit; got ${limited.length}")
+ assert(limited.forall(_.getLong(0) >= 50L),
+ s"every row must satisfy the storage filter; got ${limited.map(_.getLong(0)).toSeq}")
+ }
+ }
+ }
+
+ // ----- Partially-missing key columns, end to end through the reader -----
+
+ test("one of two key columns missing from a file: splicing runs on the values the scan returns") {
+ // This is the most intricate branch of initializeLateMaterialization, where splicing engages
+ // with a predicate that has one Literal substituted and one BoundReference renumbered, the
Review Comment:
Both fixed. The test comment says the missing key's slot points at the constant vector `initBatch` built for it, and the other says the predicate must see 7 for `b`.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2755 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.SparkException
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = readerFn(file).asInstanceOf[Iterator[Object]].map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter =
+ keyTypeAtLeastFilter(Literal(threshold), LongType, metrics)
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ // Reads a key-type fixture, returning the survivor keys and the reader. The `extract` function
+ // pulls one value at a time from the batch's key column.
+ private def readKeysAll[T](
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T,
+ capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter, (batch, i) => extract(batch.column(0), i),
+ capacity)
+
+ // Builds a `k >= threshold` storage filter bound to position 0 against a `(k, v)` schema whose
+ // key has the given type.
+ private def keyTypeAtLeastFilter(
+ threshold: Literal,
+ keyType: DataType,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, keyType, nullable = false), threshold)
+ val requested = StructType(Seq(
+ StructField("k", keyType, nullable = false), StructField("v", StringType, nullable = true)))
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("mixed: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ // Output is exact. VectorizedColumnReader uses PageReadStore.getRowIndexes (driven by
+ // our finalRanges) to skip rows within partial pages, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted has
+ // to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(195L))
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs the reader's own initialization, and it
+ // used to run that a second time on top. A reader that declined the filter on the first pass,
+ // here because every projected column is a key column, then reached the filter it had just
+ // released and failed with an NPE instead of reading plainly.
+ withTempDir { dir =>
+ val keys = (1L to 20L)
+ val path = writeKeyParquetFile(dir, keys)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(15L))),
+ requested)
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ Utils.tryWithResource(new LateMaterializationParquetRecordReader(false, 4096, filter)) {
+ reader =>
+ reader.initialize(split, context)
+ reader.initBatch(new StructType(), null)
+ val read = mutable.ArrayBuffer[Long]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ assert(read == keys, s"every row must come back unfiltered; got ${read.size}")
+ }
+ }
+ }
+
+ test("multi-batch emit: survivor count exceeds capacity") {
+ // Drive the reader at capacity = 16 with a row group of 71 survivors out of 100. Exercises:
+ // - The per-key-column queue holding multiple full-capacity vectors plus a partial tail.
+ // - The published queue head getting closed at the start of every subsequent emit.
+ // - The batch's key slots being rewritten ceil(71/16) = 5 times.
+ // The filter keeps most of the row group rather than all of it, so a reader that declined the
+ // filter altogether would return more rows than this asserts rather than passing.
+ withTempDir { dir =>
+ val keys = (1L to 100L)
+ // Big rowGroupSize so all 100 rows fit in one row group.
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 64 * 1024L)
+ val filter = keyTypeAtLeastFilter(Literal(30L), LongType)
+ val (result, reader) =
+ readKeysAll(path, filter, (vec, i) => vec.getLong(i), capacity = 16)
+ try {
+ assert(result == keys.filter(_ >= 30L).toSeq,
+ s"expected the surviving keys in order across multiple batches; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("int key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the IntegerType branch of ValueCopier.
+ withTempDir { dir =>
+ val keys = (1 to 100)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyTypeAtLeastFilter(Literal(90), IntegerType)
+ val (result, reader) = readKeysAll(path, filter, (vec, i) => vec.getInt(i))
+ try {
+ assert(result.toSet == keys.filter(_ >= 90).toSet,
+ s"expected int keys >= 90; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("string key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the StringType branch of ValueCopier (variable-length byte copy via putByteArray).
+ withTempDir { dir =>
+ val keys = (1 to 20).map(i => f"k$i%03d")
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyTypeAtLeastFilter(
+ Literal.create("k015", StringType), StringType)
+ val (result, reader) =
+ readKeysAll(path, filter, (vec, i) => vec.getUTF8String(i).toString)
+ try {
+ assert(result.toSet == keys.filter(_ >= "k015").toSet,
+ s"expected string keys >= 'k015'; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(0L))),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ Literal(bloomBytes(BloomFilter.create(10))),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // Serializes a [[BloomFilter]] to bytes suitable for a [[Literal]].
+ // The same serialization a runtime bloom's Literal carries, so these tests build theirs the way
+ // the planner does.
+ private def bloomBytes(bf: BloomFilter): Array[Byte] = BloomFilterAggregate.serialize(bf)
+
+ // Computes `XxHash64(v)` using the same seed Spark uses for runtime bloom filters.
+ private def xxHash64(v: Any, dt: DataType): Long = {
+ new XxHash64(Seq(Literal(v, dt))).eval(InternalRow.empty).asInstanceOf[Long]
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = StructType(Seq(
+ StructField("k", decimalType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = true)))
+ val key = ThrowsCheckedOn(BoundReference(0, LongType, nullable = false), bad = 7L)
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // Returns the FileSourceScanExec for a parquet read, with `storageFilters` attached. Goes through
+ // Spark's normal planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind, the SQL conf check,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def scanWithStorageFilter(
+ path: String,
+ keyName: String,
+ threshold: Long): FileSourceScanExec = {
+ val df = spark.read.parquet(path).select("k", "v")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ val keyAttr = scan.output.find(_.name == keyName).getOrElse(fail(s"No $keyName in scan output"))
+ val expr = GreaterThanOrEqual(keyAttr, Literal(threshold))
+ scan.copy(storageFilters = Seq(expr))
+ }
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect().map(r => (r.getLong(0), r.getString(1)))
+ }
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 195L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sortBy(_._1)}; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. Phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ assert(scan.simpleString(200).contains("GreaterThan(v,"),
+ s"the data filter must actually be pushed for this test to mean anything: " +
+ scan.simpleString(200))
+ // Storage filter on `k` (key column).
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(
+ storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(100L))))
+
+ val collected = executePlanCollect(withSF).toSet
+ // Every row satisfies v > 'v_000', so the storage filter alone decides the result.
+ val expected = rows.filter(_._1 >= 100L).toSet
+ assert(collected == expected,
+ s"got ${collected.toSeq.sortBy(_._1)}; expected ${expected.toSeq.sortBy(_._1)}")
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms >= 1,
+ s"expected >= 1 bloom filter on scan.storageFilters; got $storageBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSet == Set(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves bloom filter as post-scan FilterExec when conf is off") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters when conf is off; got $storageBlooms")
+ assert(postScanBlooms >= 1,
+ s"expected bloom in post-scan FilterExec when conf is off; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("the feature still engages when ignoreCorruptFiles is on") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSet == Set((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since the
+ // post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty).getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+
+ test("a row group over the splice cap is read the plain way, same rows, more bytes") {
+ // Past the cap a row group is read the plain way. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Two things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. And
+ // the fallback must actually have happened, which is observable. It reads the key column a
+ // second time, so it transfers strictly more bytes. Without that second assertion the test
+ // would pass even if the cap never reached the reader.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group read the plain way past the cap still
+ // has to emit in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") {
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, fileSchema, new StructType(), fileSchema, Nil, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), countingHadoopConf(), metricMap())
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+ readCounting(path, readerFn)
+ }
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the
+ // row ranges. `k >= 350` keeps a contiguous run, so there is one range to hold.
+ val (plainRows, plainBytes) = run("100b")
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true))),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity = 16)
+ try {
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected a row group with no survivor at all; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group past the cap charges its second key read against the byte metric") {
+ // Giving splicing up means phase 2 reads the key columns a second time, while the baseline
+ // counts them once, in phase 1. That extra read is a cost against the saving, so the same file
+ // and filter must report less avoided than they do while splicing. Without that term the metric
+ // would credit the feature with bytes it did transfer.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+
+ def avoidedBytes(cap: Long, threshold: Long): Long = {
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))),
+ fileSchema, m, cap)
+ val (_, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => b.column(0).getLong(i), capacity = 16)
+ try m.bytesAvoidedByPageFiltering.value finally reader.close()
+ }
+
+ val spliced = avoidedBytes(64L * 1024 * 1024, 350L)
+ val plain = avoidedBytes(100L, 350L)
+ assert(spliced > 0, s"page filtering has to avoid something here; got $spliced")
+ assert(plain < spliced,
+ s"the second key read must count against the saving; plain=$plain spliced=$spliced")
+
+ // The two halves of the budget are weighed together, and this is the case that says so. Eight
+ // survivors of a long key buffer 8 * (1 + 8) = 72 bytes and fall in one range of 40, so
+ // neither half reaches the 100-byte cap on its own while the sum passes it at the seventh.
+ // Weighing them separately would keep splicing here, and report the larger saving for it.
+ val eightSpliced = avoidedBytes(64L * 1024 * 1024, 393L)
+ val eightPlain = avoidedBytes(100L, 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+ }
+ }
+
+ // Writes a parquet file the low-level way, with no offset index for any column.
+ private def writeParquetFileWithoutOffsetIndex(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]]): String =
+ writeParquetFileByHand(dir, blocks, (_, _) => false)
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The three row groups are ordered so that
+ // the file's first partial read is what discovers the missing index, and the two row groups
Review Comment:
Rewritten. The header says the footer check sees the missing index for every row group before anything is read, so no read fails in this test, and it points at the damaged-index test for the retry.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReaderBenchmark.scala:
##########
@@ -91,7 +91,7 @@ object VectorizedRleValuesReaderBenchmark extends BenchmarkBase {
state
}
- // State variant with a fresh row-index iterator. `rowRanges` inside ParquetReadState is
+ // State variant with a fresh row-index iterator. The row indexes inside ParquetReadState are
Review Comment:
The numbers did not back the claim. Regenerated on GitHub Actions, they could not be compared at all, since the JDK 17 and 25 runners were 5% to 15% slower on the groups this PR does not touch. A same-machine A/B on JDK 21 then put the row-index-filtered cases 2% to 8% slower than master.
The cost was the lazy coalescing. The row-index state builds its ranges up front again, as master does, and the same A/B now puts those cases at 1.00x to 1.02x of master. The route behaves as it did before, so the description no longer claims anything about it. The benchmark file and its results are unchanged from master, and the comment at L382 is accurate again, since the field it names is `rowRanges`.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled when the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ *
+ * <p>That happens only before the first row group is loaded. The plain path reads through
+ * parquet's own row-group cursor, and the phases below read row groups by index, which never
+ * advances it. So nulling this later would restart the read at the first row group.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch's key slots hold the head of each survivor queue, not the persistent key vectors, so
+ // closing that batch would leave those unreached. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, and the plain read path skips them too. Past here one
+ // would reach `pushedFilterRangesFor`, whose `RowRanges.createSingle(0)` under
+ // `pageIndexDisabled` builds Range(0, -1) and trips its `from <= to` assertion.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Neither depends on the columns
+ // requested, so giving the filter up would only meet the same null in phase 2.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ long keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // The same ranges as the baseline when nothing was pruned, and then the walk would arrive
+ // at the same number a second time.
+ long phase2Bytes = finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
Review Comment:
Done as you write it. Phase 2 is measured over `spliceCurrentRowGroup ? nonKeyPaths : requestedPaths`, and the baseline shortcut is gated on `spliceCurrentRowGroup`, for the reason you give. `keyOnlyPaths` and its init are gone.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled when the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ *
+ * <p>That happens only before the first row group is loaded. The plain path reads through
+ * parquet's own row-group cursor, and the phases below read row groups by index, which never
+ * advances it. So nulling this later would restart the read at the first row group.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
Review Comment:
Removed. `suppliesOwnVector` returns false when the row group is not spliced and otherwise looks for the slot among `keyColumns`. The javadoc that said emit reads the array is gone with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5999523508
Thanks @peter-toth ! Will take a close look today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4187890052
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A byte child sized up front is
+ * charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}. A
+ * fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // dictionary and decoders of their last page.
+ closeDataPages();
+ releaseRowGroupReaders();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (reportedMessages.add(OFFSET_INDEX_UNREADABLE)) {
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn(OFFSET_INDEX_UNREADABLE, MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_DECODING, true);
+ // The failed decode may have grown the scratch vectors, and phase 2 reads the same pages
+ // into vectors of its own. So capacity-sized ones replace them, and beside a plain read's
+ // own vectors the reader then holds one capacity-sized batch of key vectors.
+ closeAll(keyScratchVectors);
+ allocateKeyScratch();
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = predicate.eval(keyScratchBatch.getRow(r));
+ } catch (Throwable t) {
+ // Fall back on anything but a task kill or a fatal error. The predicate runs here without
+ // the conjuncts before it in the plan, so it can throw on a row a plain scan never
+ // evaluates it on (see `ParquetStorageFilter.isSupportedStorageFilter`). Giving the
+ // filter up returns every row the pushed filter kept, and the post-scan Filter decides
+ // them in its own order, raising the same error for a row it does evaluate.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK exception
+ // (`timestamp_seconds` of a decimal) or an `AssertionError`, and a Hive UDF throws a
+ // checked SparkException. Rethrowing would fail a query a plain read answers, or drop
+ // the rest of the file silently under `ignoreCorruptFiles`.
+ //
+ // A fatal error goes out even on a row an earlier conjunct of the plan would have
+ // rejected, so the reader never runs on after one the executor would take as fatal.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_EVALUATION, false);
+ return null;
+ }
+ if (!survives) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ if (currentKeyAccumulators == null) {
+ splicedBytes +=
+ allocateKeyAccumulators(cap - splicedBytes - rangeBytes, remaining - r);
+ }
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // The buffer is this reader's own and optional, so failing to fill it gives splicing up
+ // rather than the filter, and the memory goes back at once. What fails here:
+ // - a survivor vector that cannot grow;
+ // - a key value the predicate did not read on this row, such as `b` in
+ // `coalesce(a, b)`, since the copy decodes every key;
+ // - a bug of ours, which only the WARN then shows.
+ // Phase 2 reads the key columns the way a plain read does, so a value that cannot be
+ // decoded fails the read only where a plain read fails.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together.
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, and returns the charge for the
+ * byte children it sized up front. The rest of the set is charged per row as survivors arrive.
+ * A variable-length key's byte child is reserved for what the set queued before it held, scaled
+ * to the {@code rowsLeft} rows that can still reach this set, where that is more than its default
+ * allocation. So it does not grow one doubling at a time for every capacity's worth of
+ * survivors. With its default doubling, {@code reserve} takes twice what it is asked for, so
+ * with its null bytes the child takes the factor's four bytes per value byte. That happens only
+ * when the charge and the fixed part of those rows fit within {@code headroom}, what the cap
+ * leaves. The memory is taken before any value arrives, so it is charged now, as if those values
+ * had arrived, and the values written into it use that charge up before they add to it.
+ */
+ private long allocateKeyAccumulators(long headroom, long rowsLeft) {
+ WritableColumnVector[] previous = survivorBatches.peekLast();
+ int rows = (int) Math.min(capacity, rowsLeft);
Review Comment:
Thanks, confirmed with your fixture: caps of 32216 to 32221 gave splicing up, while 32215 and 32222 kept it. Both are done.
- A later set is now sized for the survivors it can expect, at the rate seen since the row group's first survivor. I count from the first survivor rather than from the row group's start, so the rows a clustered key leaves before its range do not dilute the rate. That keeps the two re-pinned tests sized for their two values.
- The check also holds a range for each row the set is sized for.
Two new tests pin them. Your scattered fixture now splices at every cap from the smallest that splices. And a sparse last set is sized for its one survivor rather than as a full set. The fixed-part test's thresholds moved by the 80 range bytes, to 24222 and 24221.
One case stays, and the javadoc now says so. A set sized for longer values than the ones that then arrive keeps the unused part charged, since its child holds that memory. So a larger cap can still give splicing up where a smaller one, which grew the set instead, keeps splicing. With four 1000-byte keys followed by eight 10-byte ones at capacity 4, caps of 32336 to 32403 do that, while 32335 and 32404 splice. It costs one more read of the key columns, never a different answer.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import scala.util.control.NonFatal
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while a code-generated `created`
+ // holds the deserialized filter instead. Dropping the expression lets a columnar scan's task
+ // reclaim those bytes for the rest of its life. An interpreted `created`, the fallback when
+ // codegen fails or is off, keeps the expression itself, so the bytes stay. A row-based scan
+ // keeps them anyway, since its closure captures the scan node, whose storage filters hold the
+ // same bytes.
+ // Safe because every task attempt deserializes its own copy from the driver's bytes, and this
+ // lazy val is the only reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while applying a storage filter, if it must not fall back on
+ * it by giving the filter up. `fromRead` says the error came from decoding the key pages rather
+ * than from evaluating the filter. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped when the error is one the executor's kill case takes,
+ * see [[throwIfKilled]]. The error may be the kill's interrupt surfacing through a UDF that
+ * wrapped it. `PythonRunner` reads a kill off the task context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had a plain plan raised `e`, goes out as itself, so the executor finds
+ * it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in an `UnknownError`, a fatal error of another type.
+ * `FileScanRDD` would take a bare `InternalError` from a reader for a corrupt file under
+ * `ignoreCorruptFiles`. A `NonFatal` wrapper does not work either. With a kill pending, the
+ * executor reports one as killed and keeps running. A plain plan raises it from the post-scan
+ * Filter, where neither happens, and the executor stops. It stops on the `UnknownError` too, at
+ * any `spark.executor.killOnFatalError.depth` of 1 or more, since that is at the top.
+ *
+ * An error from decoding differs in three ways:
+ * - a plain read raises it from the reader, where `FileScanRDD` wraps a `NonFatal` one in
+ * FAILED_READ_FILE before the executor looks, so a fatal error deeper in the chain has to sit
+ * one level shallower to count. Under `ignoreCorruptFiles` a plain read may skip the rest of
+ * the file on it instead.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ *
+ * Anything else falls back. An evaluation error is the filter's own, which the post-scan Filter
+ * raises again for a row it does evaluate. A decoding error is met again by the plain read of
+ * the same pages.
+ */
+ def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new UnknownError("A storage filter met a fatal error it must not absorb")
+ .initCause(internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ if (Executor.isFatalError(e, if (fromRead) fatalErrorDepth - 1 else fatalErrorDepth)) {
Review Comment:
Done as you suggest, the depth is one level shallower only for a `NonFatal` decoding error. The policy test now raises your `ExceptionInInitializerError(RuntimeException(OutOfMemoryError))` from a decode at a depth of 3 and asserts the OOM goes out, and that one level further down it falls back. The scaladoc says which errors `FileScanRDD` wraps.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
+ try {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ rethrow(under(2, oom), fromRead = false)
+ } finally {
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, depth.toString)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ rethrow(new InternalError("codec"), fromRead = true)
+ rethrow(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ rethrow(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ rethrow(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val withDefault = new MetadataBuilder()
+ .putString(ResolveDefaultColumns.EXISTS_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .putString(ResolveDefaultColumns.CURRENT_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .build()
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = true, withDefault), StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split") {
+ // Two row groups give the filter up on an evaluation error, and the reader reports that once.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val rowGroups = footerOf(path).getBlocks.asScala.map(_.getRowCount)
+ assert(rowGroups.size == 2 && rowGroups.head >= 5L && rowGroups.head < 150L,
+ s"keys 5 and 150 must fall in two row groups; got $rowGroups")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val filter = createFilter(Seq(And(failingAt(5L), failingAt(150L))), kvSchema())
+ val ((result, reader), warnings) =
+ withLogged("applying it to a row raised an error")(readAll(path, filter))
+ try {
+ assert(result == rows, s"both row groups must come back whole; got ${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ val (_, sevenUncapped, _) = run("64MB", 394L)
+ val (_, sevenAtCharge, _) = run("103b", 394L)
+ val (_, sevenUnderCharge, _) = run("102b", 394L)
+ assert(sevenAtCharge == sevenUncapped,
+ s"a cap of exactly the charge must keep splicing; got $sevenAtCharge against " +
+ sevenUncapped)
+ assert(sevenUnderCharge > sevenUncapped,
+ s"a cap one byte under it must give splicing up; got $sevenUnderCharge against " +
+ sevenUncapped)
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the rows left and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the rows that can still reach it. That memory is taken before any value
+ // arrives, so it is charged then. Six survivors over a capacity of 4 fill one set of four
+ // 1000-byte keys, and the second set starts two rows before the end of the row group. So it
+ // is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000 bytes
+ // each, so they add 8000 more once the sizing is used up, and the row group holds 32142 bytes
+ // in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or a
+ // double charge would each get wrong. The second set's byte child is reserved for 2000 bytes,
+ // which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000. And
+ // the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)
+
+ val (uncappedRows, uncappedBytes) = run("64MB")
+ val (atChargeRows, atChargeBytes) = run("32142b")
+ assert(uncappedRows == 6 && atChargeRows == 6,
+ s"the filter keeps keys 15..20 either way; got $uncappedRows and $atChargeRows")
+ assert(atChargeBytes == uncappedBytes,
+ "a cap of exactly the row group's charge must keep splicing and read the key column " +
+ s"once; got $atChargeBytes against $uncappedBytes")
+ val (_, underChargeBytes) = run("32141b")
+ assert(underChargeBytes > uncappedBytes,
+ "a cap one byte under it must give splicing up and read the key column again; " +
+ s"got $underChargeBytes against $uncappedBytes")
+
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ if (cap >= 32142L) {
+ assert(secondSet == Seq(4000),
+ s"the second set must be sized for the two values left under cap $cap; got $secondSet")
+ }
+ }
+ }
+ }
+
+ test("a later accumulator set is sized only when it fits with its rows' fixed part") {
+ // The second set's two keys are 500 bytes here, shorter than the first set's 1000, so growing
+ // value by value is the cheaper path: 20142 bytes in all against 24142 sized. A cap of 24142
+ // sizes the set, whose byte child is then reserved for 2000 bytes and holds 4000. One of
+ // 24141 leaves 8033 bytes after the first set and its range, a byte short of the 8000 of
+ // sizing and the 34 of the two rows' fixed part. So the set grows instead, to a 1000-byte
+ // child, and keeps splicing. A cap of 20141 gives splicing up.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 500)
+ def run(maxSplicedBytes: String): Long =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)._2
+
+ val uncapped = run("64MB")
+ assert(run("24142b") == uncapped, "a cap that fits the sizing must keep splicing")
+ assert(run("24141b") == uncapped, "and so must one that only fits growing value by value")
+ assert(run("20141b") > uncapped, "a cap under both must give splicing up")
+
+ Seq(Long.MaxValue -> 4000, 24142L -> 4000, 24141L -> 1000).foreach { case (cap, child) =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected, s"every survivor must come back with its own key under cap $cap")
+ assert(secondSet == Seq(child),
+ s"under cap $cap the second set's byte child must hold $child bytes; got $secondSet")
+ }
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, schema, storageFilters, maxSplicedBytes, batchSize = 16)
+ val (_, uncapped) = run("64MB")
+ val (rows, atCharge) = run("166b")
+ val (_, underCharge) = run("165b")
+ assert(rows == 7, s"the filter keeps keys 394..400; got $rows")
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Two row groups of 100 rows with `v = k * 10`, and a pushed `v IN (x, 1500)`, where `x` falls
+ // between the first two pages of the first row group. Its statistics keep that row group, while
+ // its column index keeps no page of it.
+ private def writeColumnIndexEmptiedFile(dir: File): (String, Seq[sources.Filter]) = {
+ val blocks =
+ (0 until 2).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ (path, Seq(sources.In("v", Array[Any](valueBetweenFirstTwoPages(path), 1500L))))
+ }
+
+ // A value of the long column `v` that falls between the first two pages of the first row group.
+ private def valueBetweenFirstTwoPages(filePath: String): Long =
+ Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val index = reader.readColumnIndex(reader.getRowGroups.get(0).getColumns.get(1))
+ def asLong(bytes: java.nio.ByteBuffer): Long =
+ bytes.duplicate().order(java.nio.ByteOrder.LITTLE_ENDIAN).getLong(0)
+ val firstMax = asLong(index.getMaxValues.get(0))
+ val secondMin = asLong(index.getMinValues.get(1))
+ assert(secondMin - firstMax > 1,
+ s"no gap between the first two pages: $firstMax, $secondMin")
+ firstMax + 1
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize = 64L * 1024 * 1024,
+ options = Map(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT -> "100"))
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ // The rows a reader hands back before it throws, what it throws, and, for the filtering
+ // reader, whether its scratch vectors were replaced on the way.
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable, Boolean) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(false, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(false, 16, storageFilter)
+ }
+ def scratch: Seq[WritableColumnVector] = reader match {
+ case filtering: LateMaterializationParquetRecordReader =>
+ ParquetTestAccess.keyScratchVectors(filtering)
+ case _ => Nil
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val scratchBefore = scratch
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ (read.toSeq, error, scratch.zip(scratchBefore).exists { case (a, b) => a ne b })
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError, _) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError, scratchReplaced) =
+ readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ // A failed decode can leave the scratch vectors grown. Phase 2 reads the same pages into
+ // vectors of its own, so the reader replaces them with capacity-sized ones rather than keep
+ // the grown ones for the rest of the split.
+ assert(scratchReplaced, "the decoding fallback must replace the scratch vectors")
+
+ // A kill that shows only at the decoding catch, after the per-chunk check has passed, is
+ // seen there. Without that catch's own check the row group would fall back and phase 2
+ // would fail on the page instead.
+ val killedAtTheCatch = spy(TaskContext.empty())
+ doAnswer((_: InvocationOnMock) => Boolean.box(Thread.currentThread().getStackTrace
+ .exists(_.getMethodName.startsWith("rethrowIfMustPropagate"))))
+ .when(killedAtTheCatch).isInterrupted()
+ val (_, killed, _) = TaskContext.withTaskContext(killedAtTheCatch) {
+ readUntilFailure(keyAtLeastFilter(500L))
+ }
+ assert(killed.isInstanceOf[SparkException] && killed.getMessage.contains("task kill") &&
+ killed.getCause != null && killed.getCause.getClass == plainError.getClass,
+ s"the kill must go out from the decoding catch, around the decoding error; got $killed")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+
+ test("a key vector that cannot grow, or a codec's InternalError, gives the row group up") {
+ // Both are fatal-looking errors that decoding the key pages can raise, and the reader falls
+ // back on both. A scratch vector that cannot grow is phase 1's own, and phase 2 then reads
+ // what a plain read reads. An `InternalError` from decoding is how a codec like hadoop-lzo
+ // reports a corrupt page, which the plain read of the same pages meets again. Raised from the
+ // filter instead, the first goes out as its OutOfMemoryError and the second in an
+ // `UnknownError`.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ Seq(
+ new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve")),
+ new InternalError("codec")).foreach { error =>
+ val failing: KeyReaderHook.OnKeyReader = (_, keyReader) => {
+ val failingReader = spy(keyReader)
+ doThrow(error).when(failingReader).readBatch(anyInt(), any(), any(), any())
+ failingReader
+ }
+ val ((result, reader), warnings) = try {
+ withLogged("decoding its key columns raised an error") {
+ readAll(path, keyAtLeastFilter(10L), onKeyReader = failing)
+ }
+ } catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case e: VirtualMachineError => fail(s"the reader must fall back on $error; got $e", e)
+ }
+ try {
+ assert(result == rows, s"the row group must come back whole on $error; got $result")
+ assert(warnings == 1, s"the decoding fallback must report $error; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("a key value the filter does not read and cannot be decoded gives splicing up") {
+ // The filter need not read every key on every row, as `coalesce(a, b)` reads `b` only where
+ // `a` is null. The copy into the survivor buffer reads every key, so it can meet a value no
+ // plain read decodes. A dictionary-encoded column decodes a value only when it is read, so a
+ // dictionary that holds fewer values than its ids point at is such a value. Copying it gives
+ // splicing up for the row group, and phase 2 reads the key columns the way a plain read does.
+ // So the read answers where a plain one does, and the filter still applies.
+ withTempDir { dir =>
+ // Two copies of each `b`, so parquet keeps its dictionary. Cut to 64 values, it leaves `b`
+ // undecodable from row 128 on. `a` is null only before that, so the filter never reads a
+ // damaged `b`.
+ val rows = (0 until 200).map { i =>
+ (if (i < 100 && i % 5 == 0) null else s"a_$i", s"b_${i / 2}", i)
+ }
+ val path = writeSingleParquetFile(dir, rows.toDF("a", "b", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE, dictionary = true)
+ assert(encodingsOf(path, "b").exists(_.usesDictionary),
+ s"the fixture's whole point; got ${encodingsOf(path, "b")}")
+ shrinkDictionary(path, column = "b", entries = 64)
+ val columns = Seq("a", "b", "v")
+ intercept[ArrayIndexOutOfBoundsException] {
+ readAllWith(path, columns, null, (b, i) => b.column(1).getUTF8String(i).toString)._2
+ .close()
+ }
+
+ val schema = StructType(Seq(
+ StructField("a", StringType), StructField("b", StringType), StructField("v", IntegerType)))
+ val coalesced = Coalesce(Seq(BoundReference(0, StringType, nullable = true),
+ BoundReference(1, StringType, nullable = true)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(coalesced, Literal.create("a_15", StringType))), schema)
+ val ((result, reader), givenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, columns, filter, (b, i) => b.column(2).getInt(i), capacity = 16)
+ }
+ try {
+ // Keeps rows 150-199 among others, whose `b` is damaged.
+ val expected = rows.filter { case (a, b, _) => Option(a).getOrElse(b) >= "a_15" }.map(_._3)
+ assert(expected.contains(150), "the filter must keep a row whose `b` is damaged")
+ assert(result == expected, s"got ${result.size} rows, expected ${expected.size}")
+ assert(givenUp == 1,
+ "the copy must have failed and given splicing up once, or this proves nothing; " +
+ s"logged $givenUp")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry reads the damaged row group whole over the pushed data filter's ranges, and only
+ // that one, since parquet keeps a column index store per block. What it must not give up is
+ // the pushed data filter's own page pruning, which a plain read of the same query keeps, nor
+ // the other row groups' narrowing. Dropping either would read more than it needs to.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warned) = withPartialReadWarning(read(evenKeys))
+ assert(warned, "the read must have discovered the damaged index, or this test proves nothing")
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ // The row groups around the damaged one keep the storage filter's narrowing too, so they
+ // return even keys alone.
+ val undamaged = filtered.filter(k => k <= 100L || k > 200L)
+ assert(undamaged.nonEmpty && undamaged.forall(_ % 2 == 0),
+ s"the other row groups must still be narrowed; got ${undamaged.filter(_ % 2 != 0)}")
+
+ // The WARN is reported once per split, however many row groups it costs.
+ val twiceDamaged = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(twiceDamaged, blockIdx = 0, columnIdx = 0)
+ breakOffsetIndex(twiceDamaged, blockIdx = 1, columnIdx = 0)
+ val (_, warnings) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, evenKeys), twiceDamaged, ordinal = 0)
+ }
+ assert(warnings == 1, s"two damaged row groups must be reported once; got $warnings")
+ }
+ }
+
+ test("a kill that lands before an unreadable offset index is reported as a kill") {
+ // Parquet turns any IOException reading an offset index into `MissingOffsetIndexException`,
+ // a kill's interrupt included, so the catch asks for a kill before it takes a missing index.
+ // Here the kill lands while the last row group's one chunk is evaluated, after the chunk's own
+ // check, and with no row group after it. So only that catch can see it.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 2, columnIdx = 0)
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val killsAt202 = EqualTo(Remainder(
+ OnKey(BoundReference(0, LongType, nullable = false), at = 202L, killsTask = true),
+ Literal(2L)), Literal(0L))
+ val (thrown, warned) = withPartialReadWarning {
+ asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ readLongs(formatReader(schema, pushed, Seq(killsAt202)), path, ordinal = 0)
+ }
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException], s"got $thrown")
+ assert(!warned, "the kill must not be logged as an unreadable offset index")
+ }
+ }
+
+ test("a v2 file's DELTA_BYTE_ARRAY decimal column is read over scattered survivors") {
+ // Phase 2 skips rows inside a page whenever survivors scatter, where a plain read skips only
+ // when a pushed filter cuts a page. The parquet v2 writer stores a FIXED_LEN_BYTE_ARRAY
+ // decimal without a dictionary as DELTA_BYTE_ARRAY, whose reader could not skip until
+ // SPARK-59828, so this read failed where a plain read of the same file did not.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 1000).selectExpr("id AS k", "CAST(id * 1.25 AS DECIMAL(30, 2)) AS d"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.WRITER_VERSION -> "PARQUET_2_0"))
+ assert(encodingsOf(path, "d").contains(Encoding.DELTA_BYTE_ARRAY),
+ s"the fixture's whole point; got ${encodingsOf(path, "d")}")
+ val everyThirdRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(3L)), Literal(0L))
+ val schema = kvSchema(valueType = DecimalType(30, 2))
+ val (result, reader) = readAllWith(path, Seq("k", "d"),
+ createFilter(Seq(everyThirdRow), schema),
+ (b, i) => (b.column(0).getLong(i), b.column(1).getDecimal(i, 30, 2).toJavaBigDecimal))
+ try {
+ val expected = (0L until 1000L by 3L)
+ .map(k => (k, new java.math.BigDecimal(k).multiply(new java.math.BigDecimal("1.25"))
+ .setScale(2)))
+ assert(result == expected, s"got ${result.take(5)} (${result.size} rows)")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the survivor queue and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ val fileSchema = kvSchema()
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAll(path, filter, capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // Each survivor costs 9 bytes of long key and 40 of its own range. The sum passes the cap of
+ // 1024 at survivor 21, which releases the buffer, and the ranges alone at survivor 26, which
+ // gives the filter up. The capacity decides what the buffer holds when it is released. At
+ // 4096 that is one partly filled set of accumulators, and at 16 a full set in the queue as
+ // well. Both have to end with the queue empty and the filter given up. Keeping one without
+ // the other would leave phase 2 reading non-key columns while emit spliced keys from a queue
+ // holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAll(path, filter)
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSeq == Seq("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does, since the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers conjuncts under the relation's own column names") {
+ // The analyzer spells a reference the way the query did, so under case-insensitive resolution
+ // a bloom can name `_TMP_METADATA_ROW_INDEX` while the relation, and the schema the reader
+ // requests, call the column `_tmp_metadata_row_index`. The planner normalizes what it offers to
+ // the relation's names, so the format's gate sees the name the reader will see and declines.
+ // Without that the gate passed it, `ParquetStorageFilter.create` rejected it on the driver, and
+ // a query that runs with the conf off failed with it on.
+ withTempDir { dir =>
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", s"id AS $rowIndexName", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", rowIndexName, "v")
+ .queryExecution.optimizedPlan
+ def column(name: String): Attribute =
+ relation.output.find(_.name == name).getOrElse(fail(s"no $name in the relation output"))
+
+ def offered(key: Attribute): Seq[Expression] = {
+ val logical = LogicalFilter(
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ scanOf(physical).storageFilters
+ }
+
+ // As a control, an ordinary key spelled in another case is offered, under its own name.
+ val upperK = offered(column("k").withName("K"))
+ assert(upperK.size == 1, s"a bloom over `K` must reach the scan; got $upperK")
+ assert(upperK.head.references.map(_.name).toSet == Set("k"),
+ s"and it must carry the relation's name for the column; got ${upperK.head}")
+
+ val upperRowIndex =
+ offered(column(rowIndexName).withName(rowIndexName.toUpperCase(Locale.ROOT)))
+ assert(upperRowIndex.isEmpty,
+ s"a bloom over the row-index column must not be offered in any case; got $upperRowIndex")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for. A dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`, since `/` is floating-point division in Spark SQL, which would hand every
+ // key expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize, dictionary = dictionary)
+ }
+
+ // Every column chunk of the file, from its footer. The facts the tests below assert about a
+ // fixture are read off these rather than assumed from the write options.
+ private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] =
+ footerOf(filePath).getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq
+
+ private def footerOf(filePath: String): ParquetMetadata = {
+ val conf = spark.sessionState.newHadoopConf()
+ ParquetFooterReader.readFooter(
+ HadoopInputFile.fromPath(new Path(filePath), conf), ParquetMetadataConverter.NO_FILTER)
+ }
+
+ // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String, column: String = "k"): Set[Encoding] =
+ footerChunks(filePath).filter(_.getPath.toDotString == column)
+ .flatMap(_.getEncodings.asScala).toSet
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // builds the plain vectorized reader rather than the late-materialization one. Every read helper
+ // in this suite goes through here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need. The caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction. `onKeyReader`, if given, sees every key column reader phase 1 opens, see
+ // `KeyReaderHook`.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ if (storageFilter == null) {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } else if (onKeyReader != null) {
+ new KeyReaderHook(useOffHeap, capacity, storageFilter, onKeyReader)
+ } else {
+ new LateMaterializationParquetRecordReader(useOffHeap, capacity, storageFilter)
+ }
+ } { reader =>
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // A key value as a string in its *internal* representation, so the splicing path and the plain
+ // path can be compared without knowing the type and without external type conversion.
+ // `InternalRow.getAccessor` does the per-type read, which is why this takes the batch's row
+ // rather than the column vector.
+ private def renderValue(row: InternalRow, ordinal: Int, dt: DataType): String =
+ InternalRow.getAccessor(dt)(row, ordinal) match {
+ case null => "null"
+ // Two equal arrays are two objects, so the default toString would compare identities.
+ case bytes: Array[Byte] => bytes.mkString(",")
+ case value => value.toString
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path. Whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.getRow(i), 0, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage, one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why. There is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ // The byte-array copy and a later set's up-front sizing go through `putByteArray`, `getBinary`
+ // and `reserve`, which off-heap vectors implement apart. So the variable-length types run off
+ // heap too, over a capacity of 4, which gives the string and binary keys a later set to size.
+ offHeap <- if (ValueCopier.isVariableLength(dt)) Seq(false, true) else Seq(false)
+ } {
+ val encoding = (if (dictionary) "dictionary-encoded" else "plain-encoded") +
+ (if (offHeap) ", off-heap" else "")
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ // A copier that fails gives splicing up, and phase 2 then reads the keys plainly, so the
+ // rows alone would not show it.
+ val ((result, reader), splicingGivenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt),
+ capacity = if (offHeap) 4 else 4096, useOffHeap = offHeap)
+ }
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ assert(splicingGivenUp == 0, s"every $name survivor must be copied into the buffer")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // This is a regression test. TimeType is an AtomicType that passes every planning-time gate
+ // (it is batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a
+ // TIME join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // What FileSourceStrategy offers, what ParquetStorageFilter.create admits and what the reader
+ // can copy are one list, because isSupportedKeyType answers from ValueCopier. Both are asserted
+ // here. A type the copier has no case for would be a task failure rather than a planning-time
+ // rejection, and the delegation is what rules that out.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ // Throws for a type it has no copier for.
+ ValueCopier.forType(dt)
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ val e = intercept[SparkException](ValueCopier.forType(dt))
+ assert(e.isInternalError && !DataSourceUtils.shouldIgnoreCorruptFileException(e),
+ s"an unsupported key type is the reader's bug, which must fail loudly; got $e")
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("two storage filters on two key columns are both applied") {
+ // The filters are ANDed into one predicate over the key columns of both.
+ withTempDir { dir =>
+ val df = spark.range(0, 400).selectExpr(
+ "id AS k", "(id * 7) % 400 AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)),
+ LessThan(BoundReference(1, LongType, nullable = true), Literal(200L)))
+ val (result, reader) = readAllWith(path, Seq("k", "k2", "v"),
+ createFilter(storageFilters, schema),
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i), b.column(2).getUTF8String(i)
+ .toString), capacity = 16)
+ try {
+ val expected = (0L until 400L).map(i => (i, (i * 7) % 400, i.toString))
+ .filter { case (k, k2, _) => k >= 100L && k2 < 200L }
+ assert(result == expected, s"got ${result.size} rows, expected ${expected.size}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, which fixes the key row's
+ // layout. The reader pairs each key column's batch slot with its key-row position, so nothing
+ // depends on that order. Every other fixture mentions its keys in column order, and production
+ // does not promise that, since the conjuncts arrive in `afterScanFilters` order, which follows
+ // the join. So this has two key columns mentioned the other way round, and asserts both the
+ // sorted layout and that every key still comes back with its own row.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ keyAtLeast(100L))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // The emit path rewrites the key slots alone, and a partition slot is not one of them. It sits
+ // past the data columns and must keep the constant vector `initBatch` populated for it.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite. OffHeapColumnVector.close() frees
+ // the native buffer, so a read after close is a crash rather than stale data. A second close
+ // is harmless, since close() zeroes the addresses it freed. capacity = 16 over 71 survivors
+ // forces 5 emits, each closing the previous emit's dequeued key vectors. The other 29 rows
+ // are what makes a declining reader fail this.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = keyAtLeast(30L)
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (30L to 100L),
+ s"expected the 71 surviving keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // One ColumnarBatch is handed out for the whole read and its key slots are rewritten per
+ // batch, so a consumer holding on to an earlier getCurrentValue() would read the wrong
+ // vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new LateMaterializationParquetRecordReader(false, 8, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution, a column missing from the physical file -----
+
+ test("non-key column missing from a file: the evolved table reads and credits avoided bytes") {
+ // Schema evolution end to end, with the non-key column rather than the key missing from the
+ // older file. The clipped requested schema keeps the missing column, which the byte walks skip
+ // by themselves, since they go over the block's chunks and the older file has no chunk for it.
+ // What this pins is that such a read returns the right rows and still credits a saving.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // The older file has (k, v), the newer one (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSeq.sorted
+ val expected = (150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // A positive total says the byte walks ran over the older file's row groups as well.
+ // Asserting non-negativity would prove nothing. `SQLMetric.add` drops a negative and the
+ // value cannot go below zero.
+ val bytesRg = withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value + bytesPf.value > 0,
+ s"the walk must credit some avoided bytes; got rg=${bytesRg.value} pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = scanOf(spark.read.parquet(path).select("k", "v").queryExecution.executedPlan)
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Seq[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read. The rows come
+ // back sorted, so a row returned twice shows.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Seq[(Long, Option[Long])]) = {
+ val withSF = withStorageFilters(spark.sql(s"SELECT id, k FROM $table")) { attr =>
+ Seq(buildFilter(attr("k")))
+ }
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSeq.sorted
+ (withSF, collected)
+ }
+
+ test("all keys missing: an error on the constant reads the file rather than skipping it") {
+ // A file with none of the filter's key columns makes the predicate constant over it, so the
+ // reader decides once, from the values `initBatch` materialized for those columns. That
+ // evaluation can throw exactly as a row's value can. Here the older file's `k` reads as its
+ // DEFAULT, which is not a number, and the cast fails under ANSI. Failing open reads the file
+ // the way a plain scan would, so its rows survive. Skipping it would lose them, and rethrowing
+ // would fail a query a plain read answers.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true",
+ SQLConf.ANSI_ENABLED.key -> "true") {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql("ALTER TABLE evolved ADD COLUMN k STRING DEFAULT 'not-a-number'")
+ spark.sql("INSERT INTO evolved VALUES (4, '40'), (5, '50')")
+ val withSF = withStorageFilters(spark.sql("SELECT id, k FROM evolved")) { attr =>
+ Seq(GreaterThanOrEqual(Cast(attr("k"), LongType), Literal(1L)))
+ }
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val ids = collectRows(withSF).map(_.getLong(idPos)).toSeq.sorted
+ assert(ids == Seq(1L, 2L, 3L, 4L, 5L),
+ s"the older file must fail open and the newer one pass the cast; got $ids")
+ }
+ }
+ }
+
+ test("all keys missing: a checked exception on the constant reads the file") {
+ // The constant evaluation fails open under the same rule as a row's, a checked exception
+ // included. The older file's `k` reads as its DEFAULT 7, which the key expression throws on,
+ // so that file must be read the way a plain scan would. The newer file's keys evaluate
+ // normally and are rejected, since no Filter runs above this scan.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(OnKey(k, at = 7L, error = new SparkException(_)),
+ Literal(100L)))
+ assert(collected == Seq((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L))),
+ s"the older file must fail open and the newer one be filtered; got $collected")
+ }
+ }
+ }
+
+ test("a file with none of the projected non-key columns is read plainly") {
+ // The clipped requested schema keeps a column the file does not have, so a projected non-key
+ // column added after the older file was written still looks like something for phase 2 to
+ // read. Phase 2 reads nothing for it, so for that file the filter could only add cost, and the
+ // reader declines it there. With no Filter above this scan, the older file's rows come back
+ // unfiltered while the newer file's are still filtered.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTable("evolved_value") {
+ spark.sql("CREATE TABLE evolved_value (k BIGINT, v STRING) USING parquet")
+ spark.sql("INSERT INTO evolved_value VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+ spark.sql("ALTER TABLE evolved_value ADD COLUMN w STRING")
+ spark.sql("INSERT INTO evolved_value VALUES (4, 'd', 'x'), (5, 'e', 'y')")
+ val withSF =
+ scanWithStorageFilter(spark.sql("SELECT k, w FROM evolved_value"), threshold = 5L)
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val keys = collectRows(withSF).map(_.getLong(kPos)).toSeq.sorted
+ assert(keys == Seq(1L, 2L, 3L, 5L),
+ s"the older file must be read unfiltered and the newer one filtered; got $keys")
+ }
+ }
+ }
+
+ test("a file with none of a projected struct's requested fields is read plainly") {
+ // Under the legacy returnNullStructIfAllFieldsMissing, the clipped schema keeps a struct none
+ // of whose requested fields the file has, and adds no field it has, so phase 2 would read
+ // nothing for it. Only the struct's leaves are missing columns, not the struct, so the reader
+ // asks the file schema about every non-key leaf. The filter rejects every row, so a declined
+ // file comes back whole and one that is not comes back empty.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 20).selectExpr("id AS k", "named_struct('a', CAST(id AS INT)) AS s"),
+ rowGroupSize = 64 * 1024L)
+ val schema =
+ new StructType().add("k", LongType).add("s", new StructType().add("b", IntegerType))
+ val storageFilters = Seq(keyAtLeast(1000L))
+ withSQLConf(SQLConf.LEGACY_PARQUET_RETURN_NULL_STRUCT_IF_ALL_FIELDS_MISSING.key -> "true") {
+ val keys = readLongs(formatReader(schema, Nil, storageFilters), path, ordinal = 0)
+ assert(keys == (0L until 20L), s"the file must be read plainly; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Seq[(Long, Option[Long])] =
+ Seq((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test. The default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Seq[(Long, Option[Long])] = Seq((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected, s"got $collected; expected $expected")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(StorageFilterMetrics.ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Seq((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got $rejectsNull")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Seq[(Long, Option[Long])] =
+ Seq((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got $acceptsNull")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ val collected = collectRows(withSF).map(r => (r.getLong(0), r.getLong(2))).toSeq.sorted
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1))
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got ${collected.take(5)}")
+ }
+ }
+ }
+
+ test("repeated non-key columns line up with scattered survivors") {
+ // Phase 2 hands every leaf its ranges, so an array, a map and an array of structs are read
+ // with rows skipped inside their pages. A wrong skip would pair a key with another row's
+ // values, which the post-scan Filter cannot catch.
+ withTempDir { dir =>
+ val df = spark.range(0, 600).selectExpr("id AS k",
+ "sequence(0, CAST(id % 4 AS INT)) AS arr",
+ "map(CAST(id AS STRING), CAST(id AS INT)) AS m",
+ "array(named_struct('a', CAST(id AS INT)), named_struct('a', CAST(-id AS INT))) AS arrs")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(256L))
+ val toScala = CatalystTypeConverters.createToScalaConverter(df.schema)
+ def extract(b: ColumnarBatch, i: Int): Any = toScala(b.getRow(i))
+ val columns = df.schema.fieldNames.toSeq
+ val (all, plainReader) = readAllWith(path, columns, null, extract, capacity = 16)
+ plainReader.close()
+ val everyThirdRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(3L)), Literal(0L))
+ val (result, reader) = readAllWith(path, columns,
+ createFilter(Seq(everyThirdRow), df.schema), extract, capacity = 16)
+ try {
+ val expected = all.filter(_.asInstanceOf[Row].getLong(0) % 3 == 0)
+ assert(expected.size == 200, s"the plain read must see every row; got ${all.size}")
+ assert(result == expected, s"got ${result.take(3)}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF =
+ scanWithStorageFilter(spark.read.parquet(outDir).select("k", "s"), threshold = 195L)
+ val collected = collectRows(withSF)
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSeq.sorted
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k"))
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ test("a storage filter binds to the right column with partition and metadata columns in output") {
+ // `preparedStorageFilters` binds the filter's references against `output.take(requiredSchema
+ // .length)`, and `output` is the data columns, then the generated metadata columns, then the
+ // partition columns, then the constant metadata ones. A projection carrying all four is where a
+ // wrong prefix would bind the predicate to another column and drop rows the filter keeps, which
+ // the post-scan Filter cannot put back. The file's first column `a` is not projected, so the
+ // key sits at a different position in the relation's data schema than in the scan's.
+ withTempDir { dir =>
+ val outDir = new File(dir, "parted").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id * 7 AS a", "id AS k", "CONCAT('v_', id) AS v", "CAST(id % 3 AS INT) AS p")
+ .repartition(1)
+ .write
+ .partitionBy("p")
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select(
+ col("k"), col("v"), col("p"),
+ col("_metadata.row_index").as("ri"), col("_metadata.file_size").as("fs"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ // The assumption the binding rests on, stated where a change to `output` would break it.
+ assert(withSF.output.take(withSF.requiredSchema.length).map(_.name) ==
+ withSF.requiredSchema.fieldNames.toSeq,
+ s"the scan's leading attributes must be its data columns; got " +
+ s"${withSF.output.map(_.name)}")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val pPos = withSF.output.indexWhere(_.name == "p")
+ def collectAll(plan: SparkPlan): Seq[Seq[Any]] =
+ collectRows(plan).map(_.toSeq(plan.output.map(_.dataType))).toSeq
+ .sortBy(_(kPos).asInstanceOf[Long])
+ val filtered = collectAll(withSF)
+ // The same scan without the filter is the oracle, so the row indexes and file sizes come
+ // from the read rather than from arithmetic over how the partitions were written.
+ val expected = collectAll(withSF.copy(storageFilters = Nil))
+ .filter(_(kPos).asInstanceOf[Long] >= 150L)
+ assert(filtered == expected,
+ s"expected ${expected.size} rows with their own partition and metadata values; " +
+ s"got ${filtered.size}")
+ assert(filtered.map(_(pPos)).distinct.size == 3,
+ s"every partition value must still be represented; got ${filtered.map(_(pPos)).distinct}")
+ }
+ }
+ }
+
+ // ----- Planner gates -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false. It may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ //
+ // This hook answers about shapes and types alone. Whether the conjunct is deterministic and
+ // whether its references are projected columns of this scan are plan-visible, so they stay with
+ // the planner; `FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter`
+ // covers that half.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ // A cast key is supported. In ANSI mode it can throw on a row an earlier conjunct would have
+ // dropped, and the reader handles that where it arises, by giving the filter up for the row
+ // group. Declining here instead would also decline every widening cast, which is what type
+ // coercion inserts for a join between an int and a bigint column.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+
+ // The conf is read in the format rather than in the planner, and it is the per-scan question
+ // that carries it, asked once before anything per conjunct.
+ assert(!format.supportsStorageFilterPushdown(spark), "the feature is off with the conf off")
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ assert(format.supportsStorageFilterPushdown(spark), "and on with it on")
+ assert(!subclass.supportsStorageFilterPushdown(spark), "a subclass must not claim support")
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ assert(format.supportsStorageFilter(onCast),
+ "a cast key is pushed, and an evaluation error gives the row group up")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The rows are right either way, since the bloom stays in the post-scan Filter. What this gate
+ // saves is offering the scan a filter its reader cannot apply.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one. Every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSeq.sorted
+ assert(keys == (1L to 50L),
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support declines rather than delegating") {
+ // The default `FileFormat.buildReaderWithStorageFilters` answers None, and the caller falls
+ // back to the ordinary builder. That is what keeps the two builders from being able to call
+ // each other, and declining is safe because the post-scan Filter still holds the conjunct.
+ val storageFilters = Seq(keyAtLeast(1L))
+ val declined = new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration(), Map.empty)
+ assert(declined.isEmpty, "the default must not build a reader of its own")
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilterPushdown(spark),
+ "and it must not claim support either")
+ }
+
+ test("a scan whose format declines falls back to the ordinary reader and returns every row") {
+ // The check above is that the default answers None. This is the other half, that `inputRDD`
+ // then builds the ordinary reader and the scan still produces its rows. A `ParquetFileFormat`
+ // subclass is the shape that reaches it, since it inherits the override and declines by class,
+ // and its ordinary builder is the one an extension would have customized.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, threshold = 195L)
+ val declining = withSF.copy(
+ relation = withSF.relation.copy(fileFormat = new ParquetFileFormat() {})(spark))
+ val collected = executePlanCollect(declining).toSeq.sorted
+ assert(collected == rows,
+ s"a declining format reads the file plainly; got ${collected.size} of ${rows.size} rows")
+ }
+ }
+ }
+
+ Seq(false, true).foreach { aqe =>
+ test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") {
+ // AQE is on by default in production, and it is where the bloom subquery is planned by
+ // PlanAdaptiveSubqueries rather than PlanSubqueries, which is the path
+ // preparedStorageFilters' ScalarSubquery materialization depends on.
+ withBloomFilterTables {
+ val baseConf = Map(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString)
+ def run(pushdown: Boolean): (Int, Seq[(Long, Long)], Long) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, rows) = runBloomFilterJoin()
+ val excluded = rowsExcludedByStorageFilters(plan)
+ (countBloomFiltersInStorageFilters(plan),
+ rows.map(r => (r.getLong(0), r.getLong(1))).toSeq.sorted, excluded)
+ }
+ val (offFilters, off, _) = run(false)
+ val (onFilters, on, excluded) = run(true)
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ // Equal results are guaranteed by the post-scan Filter whether or not the scan applied the
+ // filter, so they alone would not notice AQE dropping it on the way to the final plan.
+ assert(offFilters == 0 && onFilters == 1,
+ s"the scan must carry the filter with the conf on and not with it off; " +
+ s"got on=$onFilters off=$offFilters")
+ // Nor would the filter count notice the reader declining it, which the metrics do.
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+
+ test("canonicalization keeps a storage-filter scan distinct from a plain one") {
+ // storageFilters is in doCanonicalize and in the case class equality, which is what stops
+ // exchange and subquery reuse from serving one scan's result to the other. Reuse compares
+ // canonicalized plans, so a scan that filters must not match a plain scan of the same file, and
+ // two scans carrying the same filter must still match.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, threshold = 25L)
+ val plain = withSF.copy(storageFilters = Nil)
+ assert(withSF != plain, "case class equality must take storageFilters into account")
+ assert(!withSF.sameResult(plain),
+ s"a filtering scan must not be reusable as a plain one:\n${withSF.canonicalized}\n" +
+ s"${plain.canonicalized}")
+ // A second, independently planned scan of the same file with the same filter still
+ // matches, so reuse is not disabled wholesale. Its key attribute carries a different
+ // exprId, which is what canonicalization normalizes away.
+ val sameSF = scanWithStorageFilter(path, threshold = 25L)
+ assert(withSF.sameResult(sameSF),
+ s"two scans with the same storage filter must stay reusable:\n" +
+ s"${withSF.canonicalized}\n${sameSF.canonicalized}")
+ val otherSF = scanWithStorageFilter(path, threshold = 30L)
+ assert(!withSF.sameResult(otherSF), "a different threshold is a different result")
+ }
+ }
+ }
+
+ test("whole-stage codegen off: the row-at-a-time path still applies the extracted bloom") {
+ // The planner's extraction gate is `fileFormat.supportBatch`, which does not look at
+ // whole-stage codegen, while `FileSourceScanExec.supportsColumnar` does. So with codegen off
+ // the bloom is still extracted, `returningBatch` is false, and the reader serves the spliced
+ // batch one row at a time. That is the only combination where the planner's gate is weaker
+ // than the runtime's.
+ withBloomFilterTables {
+ val baseConf = Map(
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "false")
+ def run(pushdown: Boolean): (Seq[(Long, Long)], Int, Boolean, Long) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, result) = runBloomFilterJoin()
+ val scans = collect(plan) { case s: FileSourceScanExec => s }
+ (result.map(r => (r.getLong(0), r.getLong(1))).toSeq.sorted,
+ countBloomFiltersInStorageFilters(plan),
+ scans.nonEmpty && scans.forall(!_.supportsColumnar),
+ rowsExcludedByStorageFilters(plan))
+ }
+ val (off, _, _, _) = run(false)
+ val (on, storageBlooms, noColumnarScan, excluded) = run(true)
+ assert(storageBlooms >= 1, s"the bloom must still be extracted with codegen off; " +
+ s"got $storageBlooms")
+ assert(noColumnarScan, "with codegen off no scan should output columnar batches")
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ // The assertions above hold whether or not the row path honored the filter, since the
+ // post-scan Filter answers the query either way. The metrics are what say it did.
+ assert(excluded > 0, s"the row path must have excluded rows; the metrics say $excluded")
+ }
+ }
+
+ test("off-heap: survivor, scratch and persistent vectors are freed") {
+ // A spliced batch's key vectors are freed at the next `nextBatch()`, and the reader's own
+ // vectors at `close()`. Freed off-heap memory shows as a zero address.
+ withTempDir { dir =>
+ val path = writeParquetFile(dir, (1L to 200L).map(i => (i, s"v_$i")), rowGroupSize = 256L)
+ val reader = new LateMaterializationParquetRecordReader(true, 16, keyAtLeastFilter(1L))
+ def address(vector: Any): Long =
+ vector.asInstanceOf[OffHeapColumnVector].valuesNativeAddress()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val persistentKeys = reader.resultBatch().column(0)
+ val scratch = ParquetTestAccess.keyScratchVectors(reader)
+ assert(reader.nextBatch())
+ val firstKeys = reader.resultBatch().column(0)
+ assert((firstKeys ne persistentKeys) && address(firstKeys) != 0,
+ "the first batch must be spliced, or this proves nothing")
+ assert(reader.nextBatch())
+ assert(address(firstKeys) == 0, "a spliced batch's keys must be freed by the next batch")
+ reader.close()
Review Comment:
Done. The test now keeps the published set after the second `nextBatch()` and the queued sets, through a new `ParquetTestAccess.queuedSurvivorSets`, and asserts their addresses are 0 after `close()`. It fails without `closeSplicingState()`.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
+ try {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ rethrow(under(2, oom), fromRead = false)
+ } finally {
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, depth.toString)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ rethrow(new InternalError("codec"), fromRead = true)
+ rethrow(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ rethrow(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ rethrow(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val withDefault = new MetadataBuilder()
+ .putString(ResolveDefaultColumns.EXISTS_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .putString(ResolveDefaultColumns.CURRENT_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .build()
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = true, withDefault), StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split") {
+ // Two row groups give the filter up on an evaluation error, and the reader reports that once.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val rowGroups = footerOf(path).getBlocks.asScala.map(_.getRowCount)
+ assert(rowGroups.size == 2 && rowGroups.head >= 5L && rowGroups.head < 150L,
+ s"keys 5 and 150 must fall in two row groups; got $rowGroups")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val filter = createFilter(Seq(And(failingAt(5L), failingAt(150L))), kvSchema())
+ val ((result, reader), warnings) =
+ withLogged("applying it to a row raised an error")(readAll(path, filter))
+ try {
+ assert(result == rows, s"both row groups must come back whole; got ${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ val (_, sevenUncapped, _) = run("64MB", 394L)
+ val (_, sevenAtCharge, _) = run("103b", 394L)
+ val (_, sevenUnderCharge, _) = run("102b", 394L)
+ assert(sevenAtCharge == sevenUncapped,
+ s"a cap of exactly the charge must keep splicing; got $sevenAtCharge against " +
+ sevenUncapped)
+ assert(sevenUnderCharge > sevenUncapped,
+ s"a cap one byte under it must give splicing up; got $sevenUnderCharge against " +
+ sevenUncapped)
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the rows left and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the rows that can still reach it. That memory is taken before any value
+ // arrives, so it is charged then. Six survivors over a capacity of 4 fill one set of four
+ // 1000-byte keys, and the second set starts two rows before the end of the row group. So it
+ // is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000 bytes
+ // each, so they add 8000 more once the sizing is used up, and the row group holds 32142 bytes
+ // in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or a
+ // double charge would each get wrong. The second set's byte child is reserved for 2000 bytes,
+ // which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000. And
+ // the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)
+
+ val (uncappedRows, uncappedBytes) = run("64MB")
+ val (atChargeRows, atChargeBytes) = run("32142b")
+ assert(uncappedRows == 6 && atChargeRows == 6,
+ s"the filter keeps keys 15..20 either way; got $uncappedRows and $atChargeRows")
+ assert(atChargeBytes == uncappedBytes,
+ "a cap of exactly the row group's charge must keep splicing and read the key column " +
+ s"once; got $atChargeBytes against $uncappedBytes")
+ val (_, underChargeBytes) = run("32141b")
+ assert(underChargeBytes > uncappedBytes,
+ "a cap one byte under it must give splicing up and read the key column again; " +
+ s"got $underChargeBytes against $uncappedBytes")
+
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ if (cap >= 32142L) {
+ assert(secondSet == Seq(4000),
+ s"the second set must be sized for the two values left under cap $cap; got $secondSet")
+ }
+ }
+ }
+ }
+
+ test("a later accumulator set is sized only when it fits with its rows' fixed part") {
+ // The second set's two keys are 500 bytes here, shorter than the first set's 1000, so growing
+ // value by value is the cheaper path: 20142 bytes in all against 24142 sized. A cap of 24142
+ // sizes the set, whose byte child is then reserved for 2000 bytes and holds 4000. One of
+ // 24141 leaves 8033 bytes after the first set and its range, a byte short of the 8000 of
+ // sizing and the 34 of the two rows' fixed part. So the set grows instead, to a 1000-byte
+ // child, and keeps splicing. A cap of 20141 gives splicing up.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 500)
+ def run(maxSplicedBytes: String): Long =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)._2
+
+ val uncapped = run("64MB")
+ assert(run("24142b") == uncapped, "a cap that fits the sizing must keep splicing")
+ assert(run("24141b") == uncapped, "and so must one that only fits growing value by value")
+ assert(run("20141b") > uncapped, "a cap under both must give splicing up")
+
+ Seq(Long.MaxValue -> 4000, 24142L -> 4000, 24141L -> 1000).foreach { case (cap, child) =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected, s"every survivor must come back with its own key under cap $cap")
+ assert(secondSet == Seq(child),
+ s"under cap $cap the second set's byte child must hold $child bytes; got $secondSet")
+ }
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, schema, storageFilters, maxSplicedBytes, batchSize = 16)
+ val (_, uncapped) = run("64MB")
+ val (rows, atCharge) = run("166b")
+ val (_, underCharge) = run("165b")
+ assert(rows == 7, s"the filter keeps keys 394..400; got $rows")
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Two row groups of 100 rows with `v = k * 10`, and a pushed `v IN (x, 1500)`, where `x` falls
+ // between the first two pages of the first row group. Its statistics keep that row group, while
+ // its column index keeps no page of it.
+ private def writeColumnIndexEmptiedFile(dir: File): (String, Seq[sources.Filter]) = {
+ val blocks =
+ (0 until 2).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ (path, Seq(sources.In("v", Array[Any](valueBetweenFirstTwoPages(path), 1500L))))
+ }
+
+ // A value of the long column `v` that falls between the first two pages of the first row group.
+ private def valueBetweenFirstTwoPages(filePath: String): Long =
+ Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val index = reader.readColumnIndex(reader.getRowGroups.get(0).getColumns.get(1))
+ def asLong(bytes: java.nio.ByteBuffer): Long =
+ bytes.duplicate().order(java.nio.ByteOrder.LITTLE_ENDIAN).getLong(0)
+ val firstMax = asLong(index.getMaxValues.get(0))
+ val secondMin = asLong(index.getMinValues.get(1))
+ assert(secondMin - firstMax > 1,
+ s"no gap between the first two pages: $firstMax, $secondMin")
+ firstMax + 1
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize = 64L * 1024 * 1024,
+ options = Map(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT -> "100"))
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ // The rows a reader hands back before it throws, what it throws, and, for the filtering
+ // reader, whether its scratch vectors were replaced on the way.
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable, Boolean) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(false, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(false, 16, storageFilter)
+ }
+ def scratch: Seq[WritableColumnVector] = reader match {
+ case filtering: LateMaterializationParquetRecordReader =>
+ ParquetTestAccess.keyScratchVectors(filtering)
+ case _ => Nil
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val scratchBefore = scratch
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ (read.toSeq, error, scratch.zip(scratchBefore).exists { case (a, b) => a ne b })
Review Comment:
Done, in two places. `readUntilFailure` builds both readers off heap and asserts the old scratch vectors' address is 0, which fails without the `closeAll`. A row group after the damaged one cannot run there, since the plain read of the same page fails the read. So the growth and `InternalError` test now reads two row groups off heap. The first one's decode fails, the second is filtered over the new scratch vectors, and the old ones must be freed.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
+ try {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ rethrow(under(2, oom), fromRead = false)
+ } finally {
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, depth.toString)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ rethrow(new InternalError("codec"), fromRead = true)
+ rethrow(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ rethrow(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ rethrow(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val withDefault = new MetadataBuilder()
+ .putString(ResolveDefaultColumns.EXISTS_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .putString(ResolveDefaultColumns.CURRENT_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .build()
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = true, withDefault), StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split") {
Review Comment:
Done. The test reads three row groups now. The first and the last raise the error, and the middle one drops keys 151 to 200 and raises nothing. A filter given up for the rest of the split fails it.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
+ try {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ rethrow(under(2, oom), fromRead = false)
Review Comment:
Done. A `fallsBack(e, fromRead)` helper catches `VirtualMachineError` and fails the test, and every call that expects no throw goes through it.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
Review Comment:
Done, `evaluationGivenUp` and `decodingGivenUp` sit beside the other named texts.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import scala.util.control.NonFatal
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while a code-generated `created`
+ // holds the deserialized filter instead. Dropping the expression lets a columnar scan's task
+ // reclaim those bytes for the rest of its life. An interpreted `created`, the fallback when
+ // codegen fails or is off, keeps the expression itself, so the bytes stay. A row-based scan
+ // keeps them anyway, since its closure captures the scan node, whose storage filters hold the
+ // same bytes.
+ // Safe because every task attempt deserializes its own copy from the driver's bytes, and this
+ // lazy val is the only reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while applying a storage filter, if it must not fall back on
+ * it by giving the filter up. `fromRead` says the error came from decoding the key pages rather
+ * than from evaluating the filter. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped when the error is one the executor's kill case takes,
+ * see [[throwIfKilled]]. The error may be the kill's interrupt surfacing through a UDF that
+ * wrapped it. `PythonRunner` reads a kill off the task context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had a plain plan raised `e`, goes out as itself, so the executor finds
+ * it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in an `UnknownError`, a fatal error of another type.
+ * `FileScanRDD` would take a bare `InternalError` from a reader for a corrupt file under
+ * `ignoreCorruptFiles`. A `NonFatal` wrapper does not work either. With a kill pending, the
+ * executor reports one as killed and keeps running. A plain plan raises it from the post-scan
+ * Filter, where neither happens, and the executor stops. It stops on the `UnknownError` too, at
+ * any `spark.executor.killOnFatalError.depth` of 1 or more, since that is at the top.
+ *
+ * An error from decoding differs in three ways:
+ * - a plain read raises it from the reader, where `FileScanRDD` wraps a `NonFatal` one in
+ * FAILED_READ_FILE before the executor looks, so a fatal error deeper in the chain has to sit
+ * one level shallower to count. Under `ignoreCorruptFiles` a plain read may skip the rest of
+ * the file on it instead.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ *
+ * Anything else falls back. An evaluation error is the filter's own, which the post-scan Filter
+ * raises again for a row it does evaluate. A decoding error is met again by the plain read of
+ * the same pages.
+ */
+ def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new UnknownError("A storage filter met a fatal error it must not absorb")
+ .initCause(internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ if (Executor.isFatalError(e, if (fromRead) fatalErrorDepth - 1 else fatalErrorDepth)) {
+ // The one `Executor.isFatalError` found, which no SparkOutOfMemoryError can precede.
+ val fatal = Iterator.iterate(e)(_.getCause).find(Utils.isFatalError).get
+ if (!corrupt(fatal)) throw out(fatal)
+ }
+ }
+
+ /**
+ * Whether `e` is how `WritableColumnVector.reserve` reports a vector that ran out of memory to
+ * grow, a plain `RuntimeException` caused by an `OutOfMemoryError`. The one it raises with no
+ * cause, for a capacity past what a vector can hold, has no fatal error to find, so the reader
+ * falls back on it anyway.
+ */
+ private def isVectorGrowthFailure(e: Throwable): Boolean =
+ e.getClass == classOf[RuntimeException] && e.getCause.isInstanceOf[OutOfMemoryError]
+
+ /**
+ * Throws a pending task kill, for the reader's loops that can run long without returning to
+ * `FileScanRDD`, which is where a plain read meets one.
+ */
+ def throwIfKilled(): Unit = throwIfKilled(null)
+
+ /**
+ * Throws a pending task kill, wrapped in a checked [[SparkException]] around `surfaced`, the
+ * error the kill surfaced as, if any. Under `ignoreCorruptFiles`, `FileScanRDD` would log any
+ * `RuntimeException` or `IOException` from a reader as a corrupt file and go on to open the
+ * split's remaining files, since it checks for a kill only between batches. The executor reports
+ * the task as killed either way. This covers the kills this reader sees at its own checks and
+ * catches. One that surfaces as an `IOException` from `readFilteredRowGroup` or from phase 2's
+ * decoding reaches that catch as it does for a plain read.
+ *
+ * The executor reports a task as killed only for an `InterruptedException` or a `NonFatal`
+ * error. So a `surfaced` error of any other type goes out as it is, the way a plain read raises
+ * it where it meets the same error. For example a codec's `InternalError` while decoding, or a
+ * `LinkageError` from a UDF.
Review Comment:
Agreed. It now says decoding, with a native codec's `UnsatisfiedLinkError` as the example.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A byte child sized up front is
+ * charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}. A
+ * fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // dictionary and decoders of their last page.
Review Comment:
Done as you suggest.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/StorageFilterPushdownBenchmark.scala:
##########
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import java.io.File
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetOutputFormat}
+import org.apache.parquet.hadoop.util.HadoopInputFile
+
+import org.apache.spark.SparkConf
+import org.apache.spark.benchmark.Benchmark
+import org.apache.spark.sql.{DataFrame, SparkSession}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
+import org.apache.spark.sql.catalyst.expressions.{BloomFilterMightContain, Literal, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.classic.ExpressionUtils.column
+import org.apache.spark.sql.execution.{FileSourceScanExec, SparkPlan}
+import org.apache.spark.sql.execution.datasources.parquet.StorageFilterMetrics
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.BinaryType
+import org.apache.spark.util.Utils
+
+/**
+ * Benchmark to measure storage-filter pushdown in the vectorized Parquet reader. Every case runs
+ * with `spark.sql.parquet.storageFilterPushdown.enabled` off and on, in two forms:
+ * - the join whose runtime bloom filter is the storage filter, which is the path the feature
+ * takes. The join's shuffle and sort are the same in both arms, and they grow with the rows
+ * that survive, so they dilute what the reader saves or costs;
+ * - a scan of the fact table alone under the same bloom filter, built once outside the timed
+ * body, which shows the reader's own share.
+ *
+ * Every case reads a fact table of about 100 MB in 34 row groups. The cases differ in four things:
+ * - the width of a row. One table has 50,000 rows of a long key and a 2 kB value that does not
+ * compress. The other has 1.4M rows of a long key and eight hashed long values;
+ * - the page size. Each table is written twice, in 8 kB pages and in parquet's default 1 MB
Review Comment:
Done, with your wording.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4106890425
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayOutputStream, File}
+import java.net.URI
+import java.time.LocalTime
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapreduce.Job
+import org.apache.parquet.column.Encoding
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetInputFormat, ParquetOutputFormat}
+
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, XxHash64}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FileSourceScanLike, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests the late-materialization path of [[VectorizedParquetRecordReader]] driven by a
+ * [[ParquetStorageFilter]]. Writes small multi-row-group parquet files, wires a hand-built filter
+ * into the reader, and asserts correctness + the two storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false), StructField("v", StringType, nullable = false)))
+ ParquetStorageFilter.create(Seq(expr), requested, metrics)
+ }
+
+ // Writes a single-column (just `k`) parquet file for the supplied key type via Spark's
+ // {@code Encoder}.
+ private def writeKeyOnlyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(dir, spark.createDataset(keys).toDF("k"), rowGroupSize)
+
+ // Reads a key-only file, returning the survivor keys and the reader. The {@code extract} function
+ // pulls one value at a time from the batch's key column.
+ private def readKeyOnlyAll[T](
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T,
+ capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k"), storageFilter, (batch, i) => extract(batch.column(0), i),
+ capacity)
+
+ // Builds a `k >= threshold` storage filter bound to position 0 against a key-only schema of the
+ // given key type.
+ private def keyOnlyAtLeastFilter(
+ threshold: Literal,
+ keyType: DataType,
+ metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, keyType, nullable = false), threshold)
+ val requested = StructType(Seq(StructField("k", keyType, nullable = false)))
+ ParquetStorageFilter.create(Seq(expr), requested, metrics)
+ }
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows in one row group: parquet's first row-group size check is at record 100, so
+ // `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = keyAtLeastFilter(1000L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf,
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(rgSkipped.value > 0,
+ s"expected at least one row group skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedRg.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${rowsExcludedRg.value}")
+ assert(rowsExcludedPf.value == 0,
+ s"no partial-row-group filtering expected; got ${rowsExcludedPf.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(bytesAvoidedRg.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got ${bytesAvoidedRg.value}")
+ assert(bytesAvoidedPf.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got ${bytesAvoidedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val filter = keyAtLeastFilter(0L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped, rowsExcludedWithinRowGroup = rowsExcludedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(rgSkipped.value == 0, s"nothing should be skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedPf.value == 0,
+ s"nothing should be filtered; got ${rowsExcludedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("mixed: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ // Output is exact: VectorizedColumnReader uses PageReadStore.getRowIndexes (driven by
+ // our finalRanges) to skip rows within partial pages, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it: a row group that was neither skipped whole nor emitted has
+ // to have its rows counted there.
+ assert(rowsExcludedPf.value > 0,
+ s"expected rows excluded inside a kept row group; got ${rowsExcludedPf.value}")
+ assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == rows.size,
+ s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " +
+ s"should account for all ${rows.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key-only projection: phase 2 is skipped and both byte-avoided metrics are zero") {
+ // When the projected schema contains only the bloom key, phase 2 is skipped entirely
+ // (`nonKeyColumns == null` in the reader). All output rows come from the per-key-column
+ // queues populated in phase 1. Total bytes read match the no-storage-filter path (phase 1 reads
+ // the key column once instead of phase 2 re-reading it), so both `avoided` metrics are zero:
+ // there are no non-key bytes to skip. This is the shape the design notes call the biggest win,
+ // so it must not be the shape that pays for metrics.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ // k >= 195 -> last 6 keys survive; preceding row groups skipped or page-pruned.
+ val filter = keyOnlyAtLeastFilter(Literal(195L), LongType, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf,
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i))
+ try {
+ val expected = keys.filter(_ >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact survivor keys; got ${result.sorted}, expected ${expected.toSeq.sorted}")
+ assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedRg.value > 0,
+ s"a skipped row group must count its rows too; got ${rowsExcludedRg.value}")
+ // The all-keys path takes its kept-row count from `finalRowCount` rather than from a
+ // phase-2 page store, so this identity is the only thing that checks that arithmetic.
+ assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == keys.size,
+ s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " +
+ s"should account for all ${keys.size} rows")
+ // For an all-keys projection, the no-storage-filter path would have read the same key
+ // column phase 1 reads. There are no non-key bytes to avoid; both metrics are 0.
+ assert(bytesAvoidedRg.value == 0,
+ s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedRg.value}")
+ assert(bytesAvoidedPf.value == 0,
+ s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("multi-batch emit: survivor count exceeds capacity") {
+ // Drive the reader at capacity = 16 with a row group of 100 surviving rows. Exercises:
+ // - The per-key-column queue holding multiple full-capacity vectors plus a partial tail.
+ // - pendingCloseKeyVectors getting closed at the start of every subsequent emit.
+ // - The per-emit ColumnarBatch reconstruction running ceil(100/16) = 7 times.
+ withTempDir { dir =>
+ val keys = (1L to 100L)
+ // Big rowGroupSize so all 100 rows fit in one row group.
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 64 * 1024L)
+ // Filter accepts every row so the queue is fully populated.
+ val filter = keyOnlyAtLeastFilter(Literal(0L), LongType)
+ val (result, reader) =
+ readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i), capacity = 16)
+ try {
+ assert(result == keys.toSeq,
+ s"expected all keys returned in order across multiple batches; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("int key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the IntegerType branch of ValueCopier.
+ withTempDir { dir =>
+ val keys = (1 to 100)
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyOnlyAtLeastFilter(Literal(90), IntegerType)
+ val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getInt(i))
+ try {
+ assert(result.toSet == keys.filter(_ >= 90).toSet,
+ s"expected int keys >= 90; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("string key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the StringType branch of ValueCopier (variable-length byte copy via putByteArray).
+ withTempDir { dir =>
+ val keys = (1 to 20).map(i => f"k$i%03d")
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyOnlyAtLeastFilter(
+ Literal.create("k015", StringType), StringType)
+ val (result, reader) =
+ readKeyOnlyAll(path, filter, (vec, i) => vec.getUTF8String(i).toString)
+ try {
+ assert(result.toSet == keys.filter(_ >= "k015").toSet,
+ s"expected string keys >= 'k015'; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction: extractStorageFilters pre-checks each one, and
+ // by the time create runs the conjunct is gone from the post-scan Filter, so a soft rejection
+ // would silently return rows the filter excludes. create fails instead.
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+
+ // Nothing to push: the caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+ }
+
+ // Serializes a [[BloomFilter]] to bytes suitable for a [[Literal]].
+ private def bloomBytes(bf: BloomFilter): Array[Byte] = {
+ val out = new ByteArrayOutputStream()
+ bf.writeTo(out)
+ out.toByteArray
+ }
+
+ // Computes `XxHash64(v)` using the same seed Spark uses for runtime bloom filters.
+ private def xxHash64(v: Any, dt: DataType): Long = {
+ new XxHash64(Seq(Literal(v, dt))).eval(InternalRow.empty).asInstanceOf[Long]
+ }
+
+ test("rewriteForMissingKeys: regular equi-join bloom probes the null key's hash") {
+ // Regular equi-join: the runtime bloom is BloomFilterMightContain(bloom, XxHash64(key)).
+ // XxHash64 is a HashExpression, so it is nullable = false and hashes a null input to its SEED
+ // rather than producing null. Substituting Literal(null) therefore leaves a concrete probe for
+ // `xxHash64(null)`, and whether the file is kept depends on whether that hash is in the bloom.
+ // Both directions are asserted so the outcome does not hinge on a lucky bloom miss.
+ val nullKeyHash = xxHash64(null, LongType)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+
+ def rewriteWithBloomContaining(hashes: Long*): ParquetStorageFilter = {
+ val bf = BloomFilter.create(10, 128)
+ hashes.foreach(bf.putLong)
+ val expr = BloomFilterMightContain(
+ Literal(bloomBytes(bf), BinaryType),
+ new XxHash64(Seq(BoundReference(0, LongType, nullable = true))))
+ val filter = ParquetStorageFilter.create(Seq(expr), requested)
+ filter.rewriteForMissingKeys(Array(0), Array(null))
+ }
+
+ val dropped = rewriteWithBloomContaining(xxHash64(42L, LongType))
+ assert(dropped.keyColumnIndices.isEmpty,
+ "all key positions are missing; keyColumnIndices should be empty")
+ assert(!dropped.evalAllMissing(),
+ "the null key's hash is not in the bloom, so evalAllMissing must return false " +
+ "(skip the file)")
+
+ val kept = rewriteWithBloomContaining(nullKeyHash)
+ assert(kept.evalAllMissing(),
+ "the null key's hash IS in the bloom, so evalAllMissing must return true (keep the file)")
+ }
+
+ test("rewriteForMissingKeys: missing key with an existence DEFAULT probes the default's hash") {
+ // A missing column that has a non-null existence DEFAULT is materialized by
+ // ParquetColumnVector as that default, not as null. The predicate must therefore be evaluated
+ // against the default. Evaluating against null could skip a whole file whose rows all match.
+ val defaultValue = 7L
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(defaultValue, LongType))
+ val expr = BloomFilterMightContain(
+ Literal(bloomBytes(bf), BinaryType),
+ new XxHash64(Seq(BoundReference(0, LongType, nullable = true))))
+ // Substituting the default keeps the file, because the default's hash is in the bloom.
+ assert(ParquetStorageFilter.create(Seq(expr), requested)
+ .rewriteForMissingKeys(Array(0), Array(defaultValue)).evalAllMissing(),
+ "substituting the existence default must probe the default's hash and keep the file")
+ // Substituting null instead would drop it, the bug this guards against. A second filter,
+ // because a rewrite is cached per set of missing positions: one scan always substitutes the
+ // same values for them.
+ assert(!ParquetStorageFilter.create(Seq(expr), requested)
+ .rewriteForMissingKeys(Array(0), Array(null)).evalAllMissing(),
+ "sanity check: substituting null probes a different hash and would drop the file")
+ }
+
+ test("rewriteForMissingKeys: null-safe equi-join bloom keeps the file") {
+ // Null-safe equi-join: ExtractEquiJoinKeys rewrites `a <=> b` so the join key becomes
+ // Coalesce(key, default). After rewriteForMissingKeys substitutes null for the
+ // BoundReference, the coalesce produces `default` (0L here) and the bloom probe checks
+ // whether the default's hash is in the bloom (which it is, matching what the creation side
+ // would have inserted for its own null rows). Expected result: keep the file.
+ val defaultHash = xxHash64(0L, LongType)
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(defaultHash)
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ val expr = BloomFilterMightContain(
+ bloomLit,
+ new XxHash64(Seq(Coalesce(Seq(
+ BoundReference(0, LongType, nullable = true),
+ Literal.default(LongType))))))
+
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(expr), requested)
+
+ val rewritten = filter.rewriteForMissingKeys(Array(0), Array(null))
+ assert(rewritten.keyColumnIndices.isEmpty,
+ "all key positions are missing; keyColumnIndices should be empty")
+ assert(rewritten.evalAllMissing(),
+ "Coalesce(null, default) yields default; bloom hit, so evalAllMissing must return true")
+ }
+
+ test("rewriteForMissingKeys: partial-missing keys narrow keyColumnIndices and renumber") {
+ // Two keys, one missing, one present. Verify the present key's BoundReference is renumbered to
+ // position 0 in the new layout, and the missing one is substituted with Literal(null). We don't
+ // run the predicate here, only the rewrite's structure.
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(1L, LongType))
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ // Two conjuncts: one on ordinal 0 (missing), one on ordinal 1 (present).
+ val exprs = Seq(
+ BloomFilterMightContain(
+ bloomLit, new XxHash64(Seq(BoundReference(0, LongType, nullable = true)))),
+ BloomFilterMightContain(
+ bloomLit, new XxHash64(Seq(BoundReference(1, LongType, nullable = true)))))
+
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(exprs, requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1))
+
+ val rewritten = filter.rewriteForMissingKeys(Array(0), Array(null))
+ assert(rewritten.keyColumnIndices.toSeq == Seq(1),
+ "only the present key column should remain in keyColumnIndices")
+ val ordinals = rewritten.boundExpression.collect {
+ case b: BoundReference => b.ordinal
+ }
+ assert(ordinals == Seq(0),
+ "the remaining BoundReference (for ordinal 1 in requested schema) should be " +
+ s"renumbered to local position 0; got $ordinals")
+ }
+
+ // Returns the FileSourceScanExec for a parquet read, with `storageFilters` attached. Goes through
+ // Spark's normal planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind, the SQL conf check,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def scanWithStorageFilter(
+ path: String,
+ keyName: String,
+ threshold: Long): FileSourceScanExec = {
+ val df = spark.read.parquet(path).select("k", "v")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == keyName).getOrElse(fail(s"No $keyName in scan output"))
+ val expr = GreaterThanOrEqual(keyAttr, Literal(threshold))
+ scan.copy(storageFilters = Seq(expr))
+ }
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect().map(r => (r.getLong(0), r.getString(1)))
+ }
+
+ test("end-to-end via FileSourceScanExec: conf on, filter applied, metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 195L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sortBy(_._1)}; expected $expected")
+
+ val rgSkipped = scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose: phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ assert(scan.simpleString(200).contains("GreaterThan(v,"),
+ s"the data filter must actually be pushed for this test to mean anything: " +
+ scan.simpleString(200))
+ // Storage filter on `k` (key column).
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(
+ storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(100L))))
+
+ val collected = executePlanCollect(withSF).toSet
+ // Every row satisfies v > 'v_000', so the storage filter alone decides the result.
+ val expected = rows.filter(_._1 >= 100L).toSet
+ assert(collected == expected,
+ s"got ${collected.toSeq.sortBy(_._1)}; expected ${expected.toSeq.sortBy(_._1)}")
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ plan.collect {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ plan.collect {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables (bf1: large, bf2: small with selective filter) for join tests.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 = "large" application side: 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 = "small" creation side: 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ body
+ }
+ }
+
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms >= 1,
+ s"expected >= 1 bloom filter on scan.storageFilters; got $storageBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune: the reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSet == Set(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves bloom filter as post-scan FilterExec when conf is off") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters when conf is off; got $storageBlooms")
+ assert(postScanBlooms >= 1,
+ s"expected bloom in post-scan FilterExec when conf is off; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("the feature still engages when ignoreCorruptFiles is on") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSet == Set((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since the
+ // post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty).getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+
+ test("FileSourceStrategy extraction preserves query results") {
+ withBloomFilterTables {
+ val baseConf = Map(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false")
+ val resultConfOff = withSQLConf(
+ (baseConf + (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false")).toSeq: _*) {
+ runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet
+ }
+ val resultConfOn = withSQLConf(
+ (baseConf + (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true")).toSeq: _*) {
+ runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet
+ }
+ assert(resultConfOn == resultConfOff,
+ s"results differ between conf-on and conf-off: on=$resultConfOn off=$resultConfOff")
+ }
+ }
+
+ Seq(Seq("k", "v"), Seq("k")).foreach { projection =>
+ test("a row group over the splice cap is read the plain way, same rows, more bytes " +
+ s"(projection ${projection.mkString(",")})") {
+ // Past the cap a row group is read the plain way: phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Two things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. And
+ // the fallback must actually have happened, which is observable: it reads the key column a
+ // second time, so it transfers strictly more bytes. Without that second assertion the test
+ // would pass even if the cap never reached the reader.
+ //
+ // The batch size is what makes the cap reachable at all: the reader examines the count only
+ // once a batch worth of survivors per key column has been buffered, so 51 survivors have to
+ // cross that line more than once.
+ //
+ // The `k` projection is the interesting one: an all-keys projection normally skips phase 2
+ // entirely, so past the cap it has to read the key column there like any other column.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val readSchema = StructType(projection.map(fileSchema(_)))
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, fileSchema, new StructType(), readSchema, Nil, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, Map.empty)
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = readerFn(file).asInstanceOf[Iterator[Object]].map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the
+ // row ranges: `k >= 350` keeps a contiguous run, so there is one range to hold.
+ val (plainRows, plainBytes) = run("100b")
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, fewer than the capacity, so the cap is never weighed and the row
+ // group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = ParquetStorageFilter.create(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true))),
+ StorageFilterMetrics(rowGroupsSkipped = rgSkipped),
+ maxSplicedRowGroupBytes = 100L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity = 16)
+ try {
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(rgSkipped.value >= 1,
+ s"expected a row group with no survivor at all; got ${rgSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group past the cap charges its second key read against the byte metric") {
+ // Giving splicing up means phase 2 reads the key columns a second time, while the baseline
+ // counts them once, in phase 1. That extra read is a cost against the saving, so the same file
+ // and filter must report less avoided than they do while splicing. Without that term the metric
+ // would credit the feature with bytes it did transfer.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+
+ def avoidedBytes(cap: Long): Long = {
+ val pf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = ParquetStorageFilter.create(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L))),
+ fileSchema, StorageFilterMetrics(bytesAvoidedByPageFiltering = pf), cap)
+ val (_, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => b.column(0).getLong(i), capacity = 16)
+ try pf.value finally reader.close()
+ }
+
+ val spliced = avoidedBytes(64L * 1024 * 1024)
+ val plain = avoidedBytes(100L)
+ assert(spliced > 0, s"page filtering has to avoid something here; got $spliced")
+ assert(plain < spliced,
+ s"the second key read must count against the saving; plain=$plain spliced=$spliced")
+ }
+ }
+
+ test("a row group whose survivors scatter into too many ranges is read without the filter") {
+ // `finalRanges` is built one row at a time, so an alternating filter makes one range per
+ // surviving row, and every column reader phase 2 drives materializes that list for the row
+ // group. Past the cap the filter is given up for that row group and every one of its rows is
+ // emitted, which the post-scan Filter then narrows.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def rowsWithCap(cap: Long, capacity: Int = 4096): Int = {
+ val filter =
+ ParquetStorageFilter.create(Seq(everyOtherRow), fileSchema, StorageFilterMetrics(), cap)
+ val (emitted, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.getRow(i).getLong(0), capacity)
+ try emitted.size finally reader.close()
+ }
+
+ assert(rowsWithCap(64L * 1024 * 1024) == 200,
+ "with room for the ranges, only the surviving rows come back")
+ // The budget is weighed in two places, and the capacity decides which one fires. At 4096 the
+ // 200 survivors never fill an accumulator, so only the check after phase 1 can catch them. At
+ // 16 the in-loop check is reached first, and it has to abandon the buffer and give the filter
+ // up together: keeping one without the other would leave phase 2 reading non-key columns
+ // while emit spliced keys from a queue holding only the first survivors.
+ assert(rowsWithCap(1024L) == 400,
+ "past the cap after phase 1, the filter is given up and the row group comes back whole")
+ assert(rowsWithCap(1024L, capacity = 16) == 400,
+ "and the same past the cap inside phase 1's survivor loop")
+ }
+ }
+
+ test("ANSI mode: a bloom over a cast join key stays in the post-scan Filter") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while a reader would evaluate
+ // it on every row of the ranges the pushed filter left. Extraction therefore has to decline, or
+ // this query would start failing when the conf is turned on.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it: a separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"a cast key must not be extracted.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSet == Set("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // `ParquetStorageFilter.test` evaluates the predicate without calling
+ // `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a
+ // `Nondeterministic` expression, so a non-deterministic conjunct has to stay behind. No
+ // producer builds one today, hence the hand-built plan: `InjectRuntimeFilter`'s blooms hash
+ // join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(42L, LongType))
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // Control: the same bloom over the key alone is offered to the scan, so the arms differ in
+ // exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. Two gates reject
+ // this one now: the planner's `deterministic` test, and `canEvaluateUnconditionally` inside
+ // the format's answer, whose whitelist is deterministic expressions only.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyOnlyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for: a dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`: `/` is floating-point division in Spark SQL, which would hand every key
+ // expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k"), rowGroupSize,
+ dictionary = dictionary)
+ }
+
+ // The encodings parquet actually used, over every column chunk of the file. Asserted rather than
+ // assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String): Set[Encoding] = {
+ val reader = ParquetFileReader.open(spark.sessionState.newHadoopConf(), new Path(filePath))
+ try {
+ reader.getFooter.getBlocks.asScala
+ .flatMap(_.getColumns.asScala)
+ .flatMap(_.getEncodings.asScala)
+ .toSet
+ } finally {
+ reader.close()
+ }
+ }
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // selects the plain (non-splicing) vectorized path. Every read helper in this suite goes through
+ // here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need: the caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } { reader =>
+ reader.setStorageFilter(storageFilter)
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // Renders one column value as a string using its *internal* representation, so the splicing path
+ // and the plain path can be compared without going through external type conversion.
+ private def renderValue(vec: ColumnVector, i: Int, dt: DataType): String = {
+ if (vec.isNullAt(i)) {
+ "null"
+ } else {
+ dt match {
+ case BooleanType => vec.getBoolean(i).toString
+ case ByteType => vec.getByte(i).toString
+ case ShortType => vec.getShort(i).toString
+ case IntegerType | DateType | _: YearMonthIntervalType => vec.getInt(i).toString
+ case LongType | TimestampType | TimestampNTZType | _: TimeType |
+ _: DayTimeIntervalType => vec.getLong(i).toString
+ case FloatType => vec.getFloat(i).toString
+ case DoubleType => vec.getDouble(i).toString
+ case d: DecimalType => vec.getDecimal(i, d.precision, d.scale).toString
+ case _: StringType => vec.getUTF8String(i).toString
+ case BinaryType => vec.getBinary(i).mkString(",")
+ case other => fail(s"renderValue does not handle $other")
+ }
+ }
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path: whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.column(0), i, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage: one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why: there is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ } {
+ val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded"
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyOnlyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = StructType(Seq(StructField("k", dt, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, dt))
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // Regression test: TimeType is an AtomicType that passes every planning-time gate (it is
+ // batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a TIME
+ // join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyOnlyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = StructType(Seq(StructField("k", dt, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // This is the contract that keeps FileSourceStrategy, ParquetStorageFilter.create and
+ // VectorizedParquetRecordReader.copierFor in lockstep. A type admitted here but missing from
+ // copierFor turns a planning-time rejection into a task failure.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyOnlyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("two key columns: both accumulators stay aligned with each other and with the data column") {
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300 => a in [100, 150]
+ val bound = And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)),
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1), "both key columns should be recognized")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, and the emit path depends on
+ // that: it pairs the k-th key slot of the batch with key-row position k, which is the identity
+ // only while the list is ascending. Nothing else in the suite supplies an unsorted one, because
+ // every other fixture happens to mention its keys in column order. Production does not promise
+ // that: the conjuncts arrive in `afterScanFilters` order, which follows the join. Without the
+ // sort this returns `a` and `b` swapped, with no exception to notice.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // Same predicate as the test above, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // Exercises the `i < isKeyTopLevel.length` branch of the emit loop: the partition slot sits
+ // past the end of isKeyTopLevel and must come from persistentBatchColumns, not the key queues.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite: OffHeapColumnVector.close() frees
+ // the native buffer, so a double close or a read after close is a crash rather than stale
+ // data. capacity = 16 over 100 survivors forces 7 emits, each closing the previous emit's
+ // dequeued key vectors.
+ withTempDir { dir =>
+ val path = writeKeyOnlyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(0L))
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (1L to 100L), s"expected all 100 keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test(s"$mode vectors: row-group skipping and page filtering over a mixed projection") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = keyAtLeastFilter(195L, StorageFilterMetrics(
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val reader = new VectorizedParquetRecordReader(useOffHeap, 4096)
+ reader.setStorageFilter(filter)
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ while (i < batch.numRows()) {
+ collected += ((batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+ i += 1
+ }
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 195L),
+ s"expected exact filtering; got ${collected.map(_._1)}")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and the
+ // partially kept one. Both counters must be non-negative, and together positive.
+ assert(bytesAvoidedRg.value >= 0 && bytesAvoidedPf.value >= 0,
+ s"avoided-byte metrics must never go negative; got rg=${bytesAvoidedRg.value} " +
+ s"pf=${bytesAvoidedPf.value}")
+ assert(bytesAvoidedRg.value + bytesAvoidedPf.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // The per-emit ColumnarBatch is replaced on every nextBatch(), so a consumer holding on to an
+ // earlier getCurrentValue() would read the wrong vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new VectorizedParquetRecordReader(false, 8)
+ try {
+ reader.setStorageFilter(filter)
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution: a column missing from the physical file -----
+
+ test("non-key column missing from a file: byte-avoided metrics tolerate a missing offset index") {
+ // ColumnIndexStore returns a null OffsetIndex for a column that is in the (clipped) requested
+ // schema but absent from the row group, which is exactly what schema evolution produces. The
+ // avoided-bytes walk runs over the full requested schema on every row group of every file, so
+ // without a null guard this is an NPE on the first row group of the older file.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // Older file: (k, v). Newer file: (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect()
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSet
+ val expected = ((150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))).toSet
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // The avoided-bytes counters are what walk the offset index of the missing column. Their
+ // being populated and non-negative is the evidence that the walk ran and coped.
+ val bytesRg = withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value >= 0 && bytesPf.value >= 0,
+ s"avoided-byte metrics must never go negative; got rg=${bytesRg.value} " +
+ s"pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = spark.read.parquet(path).select("k", "v").queryExecution.executedPlan
+ .collect { case s: FileSourceScanExec => s }.head
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Set[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Set[(Long, Option[Long])]) = {
+ val df = spark.sql(s"SELECT id, k FROM $table")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(buildFilter(keyAttr)))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = scan.output.indexWhere(_.name == "id")
+ val kPos = scan.output.indexWhere(_.name == "k")
+ val collected = rowPlan.executeCollect()
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSet
+ (withSF, collected)
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test: the default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Set[(Long, Option[Long])] = Set((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Set((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got ${rejectsNull.toSeq.sorted}")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got ${acceptsNull.toSeq.sorted}")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect().map(r => (r.getLong(0), r.getLong(2))).toSet
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1)).toSet
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got " +
+ s"${collected.toSeq.sorted.take(5)}")
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select("k", "s")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(195L))))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect()
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSet
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k")).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sorted}; expected $expected")
+ }
+ }
+ }
+
+ // ----- Planner gates and the lost-filter invariant -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false: it may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ assert(!subclass.supportsStorageFilter(bloom), "a subclass must not claim support")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ // A key the reader could not evaluate on rows the plan would have excluded. In ANSI mode the
+ // cast throws on a string that is not a number, and the reader evaluates the predicate on every
+ // row the pushed filter left, including those an earlier conjunct would have dropped.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+ assert(!format.supportsStorageFilter(onCast), "a cast key can throw on rows the plan excluded")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The whole lost-filter safety argument rests on this gate: if the reader cannot do late
+ // materialization, the planner must NOT move the bloom out of the post-scan Filter.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one: every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, "k", threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSet
+ assert(keys == (1L to 50L).toSet,
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support ignores a non-empty storageFilters") {
+ // The default `FileFormat.buildReaderWithStorageFilters` body may ignore what it is handed,
+ // because the post-scan Filter still holds it. Unreachable in production, since the planner
+ // asks `supportsStorageFilter` first.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(1L)))
+ val e = intercept[Exception] {
+ new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration())
+ }
+ // This stub implements no reader at all, so the error it fails with is the delegation's, which
+ // is what shows the filters were not rejected.
+ assert(e.getMessage.contains("buildReader is not supported"), e.getMessage)
+ }
+
+ Seq(false, true).foreach { aqe =>
Review Comment:
Fixed. `countBloomFiltersInStorageFilters` collects through `AdaptiveSparkPlanExec` now, using `AdaptiveSparkPlanHelper.collect`, and the AQE arm asserts the final plan's scan carries the filter with the conf on and does not with it off. You are right that equal results prove nothing there, and that is now in the test as a comment, so nobody weakens it back. c29625f
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -409,23 +622,244 @@ public boolean nextBatch() throws IOException {
}
cv.assemble();
}
- // If needed, compute row indexes within a file.
- if (rowIndexGenerator != null) {
- rowIndexGenerator.populateRowIndex(columnVectors, num);
- }
+ }
- rowsReturned += num;
+ /** Publishes {@code num} rows as the current batch. */
+ private void finishBatch(int num) {
columnarBatch.setNumRows(num);
+ rowsReturned += num;
numBatched = num;
batchIdx = 0;
- return true;
+ }
+
+ /**
+ * Points the batch's key slots at one survivor key vector each, which is what publishes them: the
+ * same {@link ColumnarBatch} is handed out every time, over an array it holds by reference. The
+ * queues keep owning those vectors until the next batch releases them, so a phase-2 read that
+ * throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void publishSurvivorKeyVectors(int num) {
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ if (keyVectorQueues[i].isEmpty()) {
+ // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop
+ // is driven by that same count. Named rather than left to NoSuchElementException.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to "
+ + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num));
+ }
+ }
+ keyVectorsPublished = true;
+ // Key slots are filled in ascending slot order while `keyIdx` walks the queues in key-list
+ // order, so the pairing is the identity only because `ParquetStorageFilter.create` sorts
+ // `keyColumnIndices` ascending. `isKeyTopLevel` says which slots are keys, not where each
+ // sits in that list, so this loop cannot re-derive the pairing: an unsorted list would swap
+ // key columns in the output batch. Only key slots are touched, since a non-key slot never
+ // holds anything but its persistent vector.
+ int keyIdx = 0;
+ for (int i = 0; i < isKeyTopLevel.length; i++) {
+ if (isKeyTopLevel[i]) spliceBatchColumns[i] = keyVectorQueues[keyIdx++].peekFirst();
Review Comment:
Done: `spliceBatchColumns[keyIndices[k]] = keyVectorQueues[k].peekFirst()`, so the pairing is read off the ordinals rather than derived from slot order, like the rest of the reader.
The out-of-order-ordinals test stays and now means something different: it proves the pairing rather than the sort. c29625f
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -151,6 +151,59 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
}
}
+ /**
+ * The conjuncts of `afterScanFilters` the file format can evaluate at the storage layer for late
+ * materialization, to prune value-column IO.
+ *
+ * They stay in the post-scan `Filter` as well, the way a pushed data filter does: the reader is
+ * offered them, not obliged to honor them, so the plan keeps the exact check. What it costs is
+ * evaluating the conjunct a second time for the rows the scan emits. What it buys is a reader
+ * free to give up on a file with no page index, or on a row group whose survivors scatter too far
+ * to hold their row ranges, without the answer depending on it.
+ *
+ * Two of the conditions are per scan, and failing either offers nothing:
+ * - The storage-filter pushdown SQL conf is on.
+ * - [[FileFormat.supportBatch]] holds for the schema the reader will see,
+ * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives
+ * its own vectorized-read decision from. Late materialization needs a batch read, so this asks
+ * about batch support rather than naming a format.
+ *
+ * The rest are per conjunct:
+ * - It is deterministic. A reader evaluates the predicate without
+ * `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a
+ * `Nondeterministic` expression, so a non-deterministic conjunct would fail at task time.
+ * - It references at least one column, and every column it references is a projected data
+ * column. A reference to something the scan does not read cannot be evaluated by the reader.
+ * - [[FileFormat.supportsStorageFilter]] accepts it. That is where the expression shapes and
+ * column types a reader can evaluate live, so this method names neither a format nor a type.
+ *
+ * One last condition is on the set that survives: at least one projected data column must be left
+ * for the reader to prune. A scan that projects nothing but the filter's own key columns reads
+ * the same columns for the same rows either way, since the reader has to read a key column to
+ * evaluate the filter on it, so offering it could only add the cost of evaluating the predicate
+ * outside the generated code. Nothing is offered in that case.
+ */
+ private def storageFiltersFor(
+ afterScanFilters: ExpressionSet,
+ fsRelation: HadoopFsRelation,
+ readDataColumns: Seq[Attribute],
+ outputDataSchema: StructType): Seq[Expression] = {
+ val sparkSession = fsRelation.sparkSession
+ val sqlConf = sparkSession.sessionState.conf
+ if (!sqlConf.parquetStorageFilterPushdownEnabled) return Nil
+ val resultSchema = StructType(fsRelation.partitionSchema.fields ++ outputDataSchema.fields)
+ if (!fsRelation.fileFormat.supportBatch(sparkSession, resultSchema)) return Nil
+
+ val dataAttrs = AttributeSet(readDataColumns)
+ val offered = afterScanFilters.filter { expr =>
Review Comment:
Right, and fixed: `afterScanFilters.toSeq.filter`, so the format is asked about the expression the reader will be given. I took the second half of your suggestion too, since the code alone does not stop a refactor from reintroducing it: the scaladoc of `supportsStorageFilter` now says `expr` is not a canonicalized form, so a format may decide by column name or field metadata. 43c328c
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Not applying the storage filter to {}: reading part of a row group needs a "
+ + "Parquet offset index, and this file was written without a page index for at least "
+ + "one projected column", e, MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is computed for a row group whose filter was given up: it read what a plain scan
+ // reads, so the answer is a certain zero. `needBytes`, not just `bytesAvoidedPf != null`,
+ // because that is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null && !filterGivenUp) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the baseline
+ // after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. `getRowRanges` is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then. It has to be
+ // guarded the same way phase 0 guards it, since it consults the pushed filter but not the
+ // conf that turns column-index filtering off.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
Review Comment:
The invariant is enforced where it is created rather than snapshotted. Phase 0 sets the full projection before it asks for the ranges, and that is the first lookup for the block, so every later lookup is over a subset. With a pushed data filter `getFilteredRecordCount()` at initialize gets there first, also with the full projection. A snapshot would not remove that requirement, it would move it: something still has to build the store with the full projection before phase 1 narrows the schema, which is what phase 0 does.
The metric walk no longer has a stake in it either. A path the store was not built with is skipped and reported once per file as an undercount, never thrown. I did write the throw first, and then removed it: `ignoreCorruptFiles` turns any exception from a reader into a silently truncated file, so a byte counter must not be able to change the answer. c29625f
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -151,6 +151,59 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
}
}
+ /**
+ * The conjuncts of `afterScanFilters` the file format can evaluate at the storage layer for late
+ * materialization, to prune value-column IO.
+ *
+ * They stay in the post-scan `Filter` as well, the way a pushed data filter does: the reader is
+ * offered them, not obliged to honor them, so the plan keeps the exact check. What it costs is
+ * evaluating the conjunct a second time for the rows the scan emits. What it buys is a reader
+ * free to give up on a file with no page index, or on a row group whose survivors scatter too far
+ * to hold their row ranges, without the answer depending on it.
+ *
+ * Two of the conditions are per scan, and failing either offers nothing:
+ * - The storage-filter pushdown SQL conf is on.
+ * - [[FileFormat.supportBatch]] holds for the schema the reader will see,
+ * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives
+ * its own vectorized-read decision from. Late materialization needs a batch read, so this asks
+ * about batch support rather than naming a format.
+ *
+ * The rest are per conjunct:
+ * - It is deterministic. A reader evaluates the predicate without
+ * `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a
+ * `Nondeterministic` expression, so a non-deterministic conjunct would fail at task time.
+ * - It references at least one column, and every column it references is a projected data
+ * column. A reference to something the scan does not read cannot be evaluated by the reader.
+ * - [[FileFormat.supportsStorageFilter]] accepts it. That is where the expression shapes and
+ * column types a reader can evaluate live, so this method names neither a format nor a type.
+ *
+ * One last condition is on the set that survives: at least one projected data column must be left
+ * for the reader to prune. A scan that projects nothing but the filter's own key columns reads
+ * the same columns for the same rows either way, since the reader has to read a key column to
+ * evaluate the filter on it, so offering it could only add the cost of evaluating the predicate
+ * outside the generated code. Nothing is offered in that case.
+ */
+ private def storageFiltersFor(
+ afterScanFilters: ExpressionSet,
+ fsRelation: HadoopFsRelation,
+ readDataColumns: Seq[Attribute],
+ outputDataSchema: StructType): Seq[Expression] = {
+ val sparkSession = fsRelation.sparkSession
+ val sqlConf = sparkSession.sessionState.conf
+ if (!sqlConf.parquetStorageFilterPushdownEnabled) return Nil
Review Comment:
The conf moved: `FileFormat.supportsStorageFilterPushdown(sparkSession)`, default false, and `ParquetFileFormat` reads the conf there together with its exact-class check, the way `supportBatch` reads Parquet confs inside the format. `FileSourceStrategy` asks it once per scan, before anything per conjunct, and names no format.
The five metrics stay for now, as a follow-up in the description. Moving the constants alone would leave the same Parquet vocabulary in `FileSourceScanLike`, only indirectly. What makes it right is a format declaring its own metrics, and the shape of that is decided by the second format to implement this. c29625f
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Not applying the storage filter to {}: reading part of a row group needs a "
+ + "Parquet offset index, and this file was written without a page index for at least "
+ + "one projected column", e, MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is computed for a row group whose filter was given up: it read what a plain scan
+ // reads, so the answer is a certain zero. `needBytes`, not just `bytesAvoidedPf != null`,
+ // because that is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null && !filterGivenUp) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the baseline
+ // after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. `getRowRanges` is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then. It has to be
+ // guarded the same way phase 0 guards it, since it consults the pushed filter but not the
+ // conf that turns column-index filtering off.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
+ // The ranges being built are about to be thrown away, so stop evaluating the rest.
+ if (filterGivenUp) return null;
+ }
+ }
+ }
+ remaining -= num;
+ }
+
+ if (accumulate) {
+ finalizePartialAccumulators();
+ }
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ // Assigned before the loop on purpose: an allocation failure part way through then leaves the
+ // vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows
+ * each.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, pushing them onto their
+ * queues once full. All key columns advance in lockstep, which is what keeps the queues aligned.
+ *
+ * <p>Returns false when the buffered survivors have passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} and this row group has
+ * given splicing up, in which case it has already released what it held and the caller must stop
+ * calling this.
+ */
+ private boolean appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ long valueBytes = 0L;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ // Measured on the destination: a dictionary-encoded source has no length of its own, since
+ // its values are read through the dictionary.
+ if (keyVariableLength[i]) valueBytes += dst.getArrayLength(dstRow);
+ }
+ }
+ splicedBytes += keyFixedBytesPerRow + valueBytes;
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ // Checked only here, so the cost is one comparison per capacity-sized vector rather than per
+ // row. A row group whose survivors fit in a single accumulator is never checked at all: it
+ // then holds one capacity-sized vector per key column, which is what a plain read holds.
+ //
+ // Two allocations share the budget, and giving splicing up releases only the first, so the
+ // cheaper concession comes first: drop the buffer, and give the filter up as well if the row
+ // ranges alone still do not fit. The second measurement is taken after that concession, so it
+ // counts the leaves phase 2 will now drive, which is every projected column rather than the
+ // non-key ones. Either way this row group stops accumulating.
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ if (splicedBytes + rowRangeStateBytes(survivorRangeCount) > cap) {
+ abandonSplicing();
+ // Splicing is already gone, so the flag is all that is left to set.
+ if (rowRangeStateBytes(survivorRangeCount) > cap) filterGivenUp = true;
+ return false;
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+ }
+ return true;
+ }
+
+ /**
+ * Gives the filter up for the row group being read, which leaves phase 2 reading every projected
+ * column over the rows the pushed data filter allowed, exactly what a plain read does. Splicing
+ * goes with it: the buffered survivors are no longer the rows that will be emitted.
+ */
+ private void giveUpFilter() {
+ filterGivenUp = true;
+ abandonSplicing();
+ }
+
+ /**
+ * Gives up splicing for the row group being evaluated and releases every survivor vector it has
+ * buffered. Phase 2 then reads the full projected schema and the emit path takes the persistent
+ * batch, so the rows are unaffected.
+ */
+ private void abandonSplicing() {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ closeAll(currentKeyAccumulators);
+ Arrays.fill(currentKeyAccumulators, null);
+ currentKeyAccumulatorRowCount = 0;
+ spliceCurrentRowGroup = false;
+ }
+
+ /**
+ * Pushes any partially-filled accumulator into its queue at row-group end so the emit path can
+ * dequeue it as the row group's final batch.
+ */
+ private void finalizePartialAccumulators() {
+ if (currentKeyAccumulatorRowCount == 0) return;
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Closes anything held by the splicing path: every vector still queued, the published head
+ * included, and partially-filled accumulators. Called from {@link #close()}.
+ */
+ private void closeSplicingState() {
Review Comment:
`closeSplicingState` calls `abandonSplicing` now, and the push-to-queue loop is one `pushAccumulatorsToQueues` that both callers go through.
The range duplication went the other way. Your first comment turned `parquet.filter.columnindex.enabled=false` into a gate on the whole feature, so the branch is gone and both sites are one `getRowRanges(blockIdx)` call, which is why there is no helper. What is left duplicated is the two-line empty-block guard, and I left that alone because each site does something different after it. 2d8d39b
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Not applying the storage filter to {}: reading part of a row group needs a "
+ + "Parquet offset index, and this file was written without a page index for at least "
+ + "one projected column", e, MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is computed for a row group whose filter was given up: it read what a plain scan
+ // reads, so the answer is a certain zero. `needBytes`, not just `bytesAvoidedPf != null`,
+ // because that is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null && !filterGivenUp) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the baseline
+ // after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. `getRowRanges` is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then. It has to be
+ // guarded the same way phase 0 guards it, since it consults the pushed filter but not the
+ // conf that turns column-index filtering off.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
Review Comment:
Two of the three channels are gone. `appendSurvivorRowToAccumulators` returns the bytes it added rather than a flag, the local `accumulate` with it, and the caller now checks `survivors == null` for the filter give-up. `survivorRangeCount` is a local of the loop that owns the budget.
The enum I did not do, because the two booleans are not redundant. `filterGivenUp` and `!spliceCurrentRowGroup` name different states and the byte metric reads them separately: a row group that gave splicing up still reports a saving, with its second key read charged against it, while one whose filter was given up reports nothing, since it read what a plain scan reads. The transitions are `giveUpFilter()` and `abandonSplicing()`, and the first calls the second, so the fourth combination cannot be built. c29625f
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:
##########
@@ -230,6 +274,34 @@ class ParquetFileFormat
val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead
val archiveFormatEnabled = parquetOptions.archiveFormatEnabled
+ // Late materialization needs the vectorized reader. `enableVectorizedReader` is recomputed from
+ // the live session conf when the RDD is built, so a flip of
+ // spark.sql.parquet.enableVectorizedReader (or the nested-column variant) after planning lands
+ // here, and the filters are simply not installed: the post-scan Filter still holds them.
+ val storageFilterOpt: Option[ParquetStorageFilter] = if (storageFilters.isEmpty) {
+ None
+ } else if (!enableVectorizedReader) {
+ logInfo(log"Not honoring storage filters for schema " +
+ log"${MDC(SCHEMA, resultSchema.catalogString)}: the vectorized Parquet reader is disabled")
+ None
+ } else {
+ val metrics = StorageFilterMetrics(
+ rowGroupsSkipped = storageFilterMetrics.getOrElse(
+ FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED, null),
+ rowsExcludedByRowGroup = storageFilterMetrics.getOrElse(
+ FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP, null),
+ rowsExcludedWithinRowGroup = storageFilterMetrics.getOrElse(
+ FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP, null),
+ bytesAvoidedByRowGroup = storageFilterMetrics.getOrElse(
+ FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP, null),
+ bytesAvoidedByPageFiltering = storageFilterMetrics.getOrElse(
+ FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING, null))
+ // `create` requires every condition extractStorageFilters already pre-checked, so a violation
Review Comment:
All fixed: `storageFiltersFor` in the three places, and the two test comments now describe the current design. The batch is one object whose key slots are rewritten in place, and `resultBatch()`'s javadoc says that, since the old comment was what round 1 flagged as a contract break. @cloud-fan flagged the same comment, so this answers both. c29625f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5776167108
#58120 has merged, so this is now rebased onto master and out of draft. Only my commit is left in the diff, and it is unchanged from the draft version, `git range-diff` reports the patch identical. `ParquetStorageFilterSuite` (89 tests) and `dev/lint-scala` pass on the new base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4084289903
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java:
##########
@@ -87,6 +87,13 @@ public abstract class SpecificParquetRecordReaderBase<T> extends RecordReader<Vo
protected ParquetRowGroupReader reader;
+ /**
+ * The opened input file and parquet footer. Stored so subclasses can read footer-derived metadata
+ * without re-opening the file. Set by both {@link #initialize} overloads.
+ */
+ protected HadoopInputFile inputFile;
Review Comment:
Dropped, both of them, and the footer is a local again. The javadoc went with them.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +929,450 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+
+ // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the
+ // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and
+ // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes
+ // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index
+ // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's
+ // ranges under the full schema, and with it off we do not call getRowRanges at all.
+ lateMatReader.setRequestedSchema(requestedSchema);
+ // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT
+ // options.useColumnIndexFilter(), so calling it unconditionally would keep applying
+ // column-index filtering after a user turned it off. That conf is the documented escape hatch
+ // for files whose column index is wrong, and trusting a wrong column index here would drop
+ // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no
+ // longer holds this predicate.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ if (pushedFilterRanges.rowCount() == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // Both byte metrics are always wired in production (FileSourceScanLike creates all five
+ // whenever storageFilters is non-empty), so this only skips the work on the test-only path
+ // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so
+ // there is nothing here to avoid on the production path.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ long baselineRows = pushedFilterRanges.rowCount();
+ long baselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema,
+ pushedFilterRanges)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyRequestedSchema);
+ long phase1Bytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema,
+ pushedFilterRanges)
+ : 0L;
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite "
+ + pushedFilterRanges.rowCount() + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+
+ if (finalRanges.rowCount() == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely. We still
+ // paid phase-1 to read the key column, so the bytes avoided vs a no-storage-filter read
+ // are baseline - phase1 (the non-key bytes the no-filter path would have read).
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(baselineBytes - phase1Bytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyRequestedSchema is null); emit reconstructs each
+ // batch from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyRequestedSchema == null) {
+ keptRows = finalRanges.rowCount();
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(nonKeyRequestedSchema);
+ // requireOffsetIndexesForPhase2() already established that every projected column of every
+ // row group has an offset index, so this page-filtering read cannot fail for want of one.
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRanges.rowCount()
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ phase2Bytes = bytesAvoidedPf != null
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, nonKeyRequestedSchema,
+ finalRanges)
+ : 0L;
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(baselineBytes - phase1Bytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the leaf columns of {@code schema} when it reads
+ * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are
+ * included, because both are read whenever any page of a chunk is read.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them. Every caller for a given block walks the same
+ * metadata, so a skipped column drops out of the baseline and the per-phase totals alike.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ MessageType schema,
+ RowRanges rowRanges) {
+ if (schema == null || rowRanges.rowCount() == 0 || schema.getColumns().isEmpty()) {
+ return 0L;
+ }
+ BlockMetaData block = reader.getRowGroups().get(blockIndex);
+ long blockRowCount = block.getRowCount();
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : block.getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ boolean wholeBlock = rowRanges.rowCount() == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : schema.getColumns()) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Reads all rows of the given key-only {@link PageReadStore} (which contains only rows in
+ * {@code pushedFilterRanges}) in capacity-sized chunks, evaluates the storage filter on each row,
+ * builds a {@link RowRanges} of surviving rows in original block-row coordinates, and appends
+ * survivor key values into the per-key-column accumulators ({@link #currentKeyAccumulators}).
+ * When an accumulator hits {@link #capacity}, it's pushed into {@link #keyVectorQueues} and a
+ * fresh one is allocated. After all rows have been examined, any partial trailing accumulator is
+ * pushed too.
+ *
+ * <p>The result is a subset of {@code pushedFilterRanges}: rows not in {@code
+ * pushedFilterRanges} were never read and are implicitly excluded.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ long keyRowsTotal = pushedFilterRanges.rowCount();
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ long remaining = keyRowsTotal;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ appendSurvivorRowToAccumulators(r);
+ }
+ }
+ remaining -= num;
+ }
+
+ finalizePartialAccumulators();
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates the per-key-column accumulator vectors if any slot is null (i.e. the previous
+ * accumulator was just pushed to the queue or this is the first row group). Each accumulator has
+ * {@link #capacity} rows.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of each {@link #keyScratchVectors} into the corresponding
+ * {@link #currentKeyAccumulators}. When the accumulators fill, they're pushed onto their queues
+ * and fresh ones allocated. All key columns are appended in lockstep so accumulators stay
+ * aligned.
+ */
+ private void appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ }
+ }
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+ }
+ }
+
+ /**
+ * Pushes any partially-filled accumulator into its queue at row-group end so the emit path can
+ * dequeue it as the row group's final batch.
+ */
+ private void finalizePartialAccumulators() {
+ if (currentKeyAccumulatorRowCount == 0) return;
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Closes anything held by the splicing path: pending dequeued key vectors not yet rolled over,
+ * any vectors still queued (e.g. on early termination), and partially-filled accumulators.
+ * Called from {@link #close()}.
+ */
+ private void closeSplicingState() {
+ if (pendingCloseKeyVectors != null) {
+ for (WritableColumnVector v : pendingCloseKeyVectors) {
+ if (v != null) v.close();
+ }
+ pendingCloseKeyVectors = null;
+ }
+ if (keyVectorQueues != null) {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ if (q != null) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ }
+ }
+ if (currentKeyAccumulators != null) {
+ for (WritableColumnVector v : currentKeyAccumulators) {
+ if (v != null) v.close();
+ }
+ currentKeyAccumulators = null;
+ }
+ }
+
+ /**
+ * Per-key-column value copier: appends one value from {@code src[srcRow]} to
+ * {@code dst[dstRow]}. Picked once at init via {@link #copierFor(DataType)}; called per surviving
+ * row in {@link #appendSurvivorRowToAccumulators}. Caller handles null sources.
+ */
+ @FunctionalInterface
+ private interface ValueCopier {
+ void copy(WritableColumnVector dst, int dstRow, WritableColumnVector src, int srcRow);
+ }
+
+ /**
+ * Returns a {@link ValueCopier} for the given key {@link DataType}. The set of types handled here
+ * is the definition behind {@code ParquetStorageFilter.isSupportedKeyType}, which gates both
+ * planning-time extraction and {@code ParquetStorageFilter.create} -- so the throw at the end is
+ * unreachable. Teach both sides at once when adding a type; a type admitted there but missing
+ * here becomes a task failure instead of a planning-time rejection.
+ */
+ private static ValueCopier copierFor(DataType dt) {
+ if (dt instanceof BooleanType) {
+ return (dst, dRow, src, sRow) -> dst.putBoolean(dRow, src.getBoolean(sRow));
+ }
+ if (dt instanceof ByteType) {
+ return (dst, dRow, src, sRow) -> dst.putByte(dRow, src.getByte(sRow));
+ }
+ if (dt instanceof ShortType) {
+ return (dst, dRow, src, sRow) -> dst.putShort(dRow, src.getShort(sRow));
+ }
+ if (dt instanceof IntegerType
+ || dt instanceof DateType
+ || dt instanceof YearMonthIntervalType) {
+ return (dst, dRow, src, sRow) -> dst.putInt(dRow, src.getInt(sRow));
+ }
+ if (dt instanceof LongType
+ || dt instanceof TimestampType
+ || dt instanceof TimestampNTZType
+ || dt instanceof TimeType
+ || dt instanceof DayTimeIntervalType) {
+ return (dst, dRow, src, sRow) -> dst.putLong(dRow, src.getLong(sRow));
+ }
+ if (dt instanceof FloatType) {
+ return (dst, dRow, src, sRow) -> dst.putFloat(dRow, src.getFloat(sRow));
+ }
+ if (dt instanceof DoubleType) {
+ return (dst, dRow, src, sRow) -> dst.putDouble(dRow, src.getDouble(sRow));
+ }
+ if (dt instanceof DecimalType decimalType) {
+ int precision = decimalType.precision();
+ if (precision <= Decimal.MAX_INT_DIGITS()) {
+ return (dst, dRow, src, sRow) -> dst.putInt(dRow, src.getInt(sRow));
+ }
+ if (precision <= Decimal.MAX_LONG_DIGITS()) {
+ return (dst, dRow, src, sRow) -> dst.putLong(dRow, src.getLong(sRow));
+ }
+ return (dst, dRow, src, sRow) -> dst.putByteArray(dRow, src.getBinary(sRow));
+ }
+ if (dt instanceof StringType
+ || dt instanceof VarcharType
+ || dt instanceof CharType
+ || dt instanceof BinaryType) {
+ return (dst, dRow, src, sRow) -> dst.putByteArray(dRow, src.getBinary(sRow));
Review Comment:
Real, and left as a follow-up named in the description. A vector-to-vector `appendBytes` means touching `WritableColumnVector` and both subclasses, which widens this change into the core column-vector classes. It is also confined to string and binary keys -- a fixed-width copier writes straight into the destination array with no allocation -- so the cost is not on every shape.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +929,450 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+
+ // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the
+ // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and
+ // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes
+ // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index
+ // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's
+ // ranges under the full schema, and with it off we do not call getRowRanges at all.
+ lateMatReader.setRequestedSchema(requestedSchema);
+ // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT
+ // options.useColumnIndexFilter(), so calling it unconditionally would keep applying
+ // column-index filtering after a user turned it off. That conf is the documented escape hatch
+ // for files whose column index is wrong, and trusting a wrong column index here would drop
+ // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no
+ // longer holds this predicate.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ if (pushedFilterRanges.rowCount() == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // Both byte metrics are always wired in production (FileSourceScanLike creates all five
+ // whenever storageFilters is non-empty), so this only skips the work on the test-only path
+ // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so
+ // there is nothing here to avoid on the production path.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ long baselineRows = pushedFilterRanges.rowCount();
+ long baselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema,
+ pushedFilterRanges)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyRequestedSchema);
+ long phase1Bytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema,
+ pushedFilterRanges)
+ : 0L;
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite "
+ + pushedFilterRanges.rowCount() + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+
+ if (finalRanges.rowCount() == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely. We still
+ // paid phase-1 to read the key column, so the bytes avoided vs a no-storage-filter read
+ // are baseline - phase1 (the non-key bytes the no-filter path would have read).
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(baselineBytes - phase1Bytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyRequestedSchema is null); emit reconstructs each
+ // batch from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyRequestedSchema == null) {
+ keptRows = finalRanges.rowCount();
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(nonKeyRequestedSchema);
+ // requireOffsetIndexesForPhase2() already established that every projected column of every
+ // row group has an offset index, so this page-filtering read cannot fail for want of one.
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRanges.rowCount()
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ phase2Bytes = bytesAvoidedPf != null
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, nonKeyRequestedSchema,
+ finalRanges)
+ : 0L;
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(baselineBytes - phase1Bytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the leaf columns of {@code schema} when it reads
+ * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are
+ * included, because both are read whenever any page of a chunk is read.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them. Every caller for a given block walks the same
+ * metadata, so a skipped column drops out of the baseline and the per-phase totals alike.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ MessageType schema,
+ RowRanges rowRanges) {
+ if (schema == null || rowRanges.rowCount() == 0 || schema.getColumns().isEmpty()) {
+ return 0L;
+ }
+ BlockMetaData block = reader.getRowGroups().get(blockIndex);
+ long blockRowCount = block.getRowCount();
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : block.getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ boolean wholeBlock = rowRanges.rowCount() == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : schema.getColumns()) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Reads all rows of the given key-only {@link PageReadStore} (which contains only rows in
+ * {@code pushedFilterRanges}) in capacity-sized chunks, evaluates the storage filter on each row,
+ * builds a {@link RowRanges} of surviving rows in original block-row coordinates, and appends
+ * survivor key values into the per-key-column accumulators ({@link #currentKeyAccumulators}).
+ * When an accumulator hits {@link #capacity}, it's pushed into {@link #keyVectorQueues} and a
+ * fresh one is allocated. After all rows have been examined, any partial trailing accumulator is
+ * pushed too.
+ *
+ * <p>The result is a subset of {@code pushedFilterRanges}: rows not in {@code
+ * pushedFilterRanges} were never read and are implicitly excluded.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ long keyRowsTotal = pushedFilterRanges.rowCount();
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ long remaining = keyRowsTotal;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ appendSurvivorRowToAccumulators(r);
+ }
+ }
+ remaining -= num;
+ }
+
+ finalizePartialAccumulators();
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates the per-key-column accumulator vectors if any slot is null (i.e. the previous
+ * accumulator was just pushed to the queue or this is the first row group). Each accumulator has
+ * {@link #capacity} rows.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of each {@link #keyScratchVectors} into the corresponding
+ * {@link #currentKeyAccumulators}. When the accumulators fill, they're pushed onto their queues
+ * and fresh ones allocated. All key columns are appended in lockstep so accumulators stay
+ * aligned.
+ */
+ private void appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ }
+ }
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+ }
+ }
+
+ /**
+ * Pushes any partially-filled accumulator into its queue at row-group end so the emit path can
+ * dequeue it as the row group's final batch.
+ */
+ private void finalizePartialAccumulators() {
+ if (currentKeyAccumulatorRowCount == 0) return;
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Closes anything held by the splicing path: pending dequeued key vectors not yet rolled over,
+ * any vectors still queued (e.g. on early termination), and partially-filled accumulators.
+ * Called from {@link #close()}.
+ */
+ private void closeSplicingState() {
+ if (pendingCloseKeyVectors != null) {
+ for (WritableColumnVector v : pendingCloseKeyVectors) {
+ if (v != null) v.close();
+ }
+ pendingCloseKeyVectors = null;
+ }
+ if (keyVectorQueues != null) {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ if (q != null) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ }
+ }
+ if (currentKeyAccumulators != null) {
+ for (WritableColumnVector v : currentKeyAccumulators) {
+ if (v != null) v.close();
+ }
+ currentKeyAccumulators = null;
+ }
+ }
+
+ /**
+ * Per-key-column value copier: appends one value from {@code src[srcRow]} to
+ * {@code dst[dstRow]}. Picked once at init via {@link #copierFor(DataType)}; called per surviving
+ * row in {@link #appendSurvivorRowToAccumulators}. Caller handles null sources.
+ */
+ @FunctionalInterface
+ private interface ValueCopier {
+ void copy(WritableColumnVector dst, int dstRow, WritableColumnVector src, int srcRow);
+ }
+
+ /**
+ * Returns a {@link ValueCopier} for the given key {@link DataType}. The set of types handled here
+ * is the definition behind {@code ParquetStorageFilter.isSupportedKeyType}, which gates both
+ * planning-time extraction and {@code ParquetStorageFilter.create} -- so the throw at the end is
+ * unreachable. Teach both sides at once when adding a type; a type admitted there but missing
+ * here becomes a task failure instead of a planning-time rejection.
+ */
+ private static ValueCopier copierFor(DataType dt) {
Review Comment:
The dead `VarcharType` and `CharType` branches are gone.
The dedup itself I would rather not do. `RowToColumnConverter` goes `InternalRow` to vector, while these copiers go vector to vector, so reusing it would put the row abstraction back in the innermost loop and, for a string, a `UTF8String` wrapper per value. That is the opposite direction from your `appendBytes` comment above, which asks the copy to get *more* specialized, and that one is the follow-up I would rather take.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +929,450 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+
+ // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the
+ // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and
+ // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes
+ // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index
+ // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's
+ // ranges under the full schema, and with it off we do not call getRowRanges at all.
+ lateMatReader.setRequestedSchema(requestedSchema);
+ // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT
+ // options.useColumnIndexFilter(), so calling it unconditionally would keep applying
+ // column-index filtering after a user turned it off. That conf is the documented escape hatch
+ // for files whose column index is wrong, and trusting a wrong column index here would drop
+ // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no
+ // longer holds this predicate.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ if (pushedFilterRanges.rowCount() == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // Both byte metrics are always wired in production (FileSourceScanLike creates all five
+ // whenever storageFilters is non-empty), so this only skips the work on the test-only path
+ // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so
+ // there is nothing here to avoid on the production path.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ long baselineRows = pushedFilterRanges.rowCount();
+ long baselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema,
+ pushedFilterRanges)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyRequestedSchema);
+ long phase1Bytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema,
+ pushedFilterRanges)
+ : 0L;
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite "
+ + pushedFilterRanges.rowCount() + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+
+ if (finalRanges.rowCount() == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely. We still
+ // paid phase-1 to read the key column, so the bytes avoided vs a no-storage-filter read
+ // are baseline - phase1 (the non-key bytes the no-filter path would have read).
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(baselineBytes - phase1Bytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyRequestedSchema is null); emit reconstructs each
+ // batch from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyRequestedSchema == null) {
+ keptRows = finalRanges.rowCount();
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(nonKeyRequestedSchema);
+ // requireOffsetIndexesForPhase2() already established that every projected column of every
+ // row group has an offset index, so this page-filtering read cannot fail for want of one.
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRanges.rowCount()
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ phase2Bytes = bytesAvoidedPf != null
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, nonKeyRequestedSchema,
+ finalRanges)
+ : 0L;
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(baselineBytes - phase1Bytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the leaf columns of {@code schema} when it reads
+ * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are
+ * included, because both are read whenever any page of a chunk is read.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them. Every caller for a given block walks the same
+ * metadata, so a skipped column drops out of the baseline and the per-phase totals alike.
+ */
+ private static long compressedBytesForRowRanges(
Review Comment:
Fixed exactly as suggested: the three `List<ColumnDescriptor>` are fields resolved once per file in `initializeLateMaterialization`, the reader is driven through the `setRequestedSchema(List<ColumnDescriptor>)` overload, the path-to-chunk map is built once per block and shared by the metric calls, and both row counts are passed in instead of recomputed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4136319944
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -80,23 +85,23 @@ public class VectorizedParquetRecordReader extends SpecificParquetRecordReaderBa
* Encapsulate writable column vectors with other Parquet related info such as
* repetition / definition levels.
*/
- private ParquetColumnVector[] columnVectors;
+ protected ParquetColumnVector[] columnVectors;
/**
* The number of rows that have been returned.
*/
- private long rowsReturned;
+ protected long rowsReturned;
Review Comment:
Done, all four are private again. `noMoreRows()` went back inline too, since the subclass no longer overrides it. So the base class has nine protected methods and five protected fields, and the subclass uses every one of them.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
Review Comment:
Added to `spark.sql.parquet.storageFilterPushdown.enabled`. It says the answer can change under `spark.sql.files.ignoreCorruptFiles`, because a corrupt page in a key column also drops the rows of its row group that a plain read returns before reaching that page. The description says the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-6001519867
@dongjoon-hyun thank you for round 10. All 15 are done in one commit, and the description is updated.
- Item 1: a later set is sized for the survivors it can expect, at the rate seen since the row group's first survivor, and only when the cap also holds a range for each of its rows. Your scattered fixture now splices at every cap from the smallest that splices.
- Item 2: the depth is one level shallower only for a `NonFatal` decoding error, and the policy test raises your `LinkageError` chain.
- The rest are tests, docs and cleanup. Each new test fails without the line it pins.
Is there anything else you would see as blocking a merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5912892579
@dongjoon-hyun thank you. Round 6 is answered in the threads, and the description is up to date.
- The row-index gate can no longer be bypassed by case. The planner normalizes the conjuncts it offers to the relation's column names.
- A task kill or a fatal error behind a key expression's error is rethrown as it is, not absorbed.
- A decode error in a key page gives the row group up, so a corrupt key page fails where a plain read fails. The conf doc describes the `ignoreCorruptFiles` difference in general terms now.
- The row-index route measured 2% to 8% slower than master, and the lazy coalescing was the cause. It builds its ranges up front again, and a same-machine A/B puts it level with master.
- Everything else is fixed as you suggested, and the new behavior has tests that fail without it.
One correction to round 5. Removing the 2-arg `initialize` override tripped MiMa, so the override is back and only calls `super`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4196655077
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/StorageFilterPushdownBenchmark.scala:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import java.io.File
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetOutputFormat}
+import org.apache.parquet.hadoop.util.HadoopInputFile
+
+import org.apache.spark.SparkConf
+import org.apache.spark.benchmark.Benchmark
+import org.apache.spark.sql.{DataFrame, SparkSession}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
+import org.apache.spark.sql.catalyst.expressions.{BloomFilterMightContain, Literal, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.classic.ExpressionUtils.column
+import org.apache.spark.sql.execution.{FileSourceScanExec, SparkPlan}
+import org.apache.spark.sql.execution.datasources.parquet.StorageFilterMetrics
+import org.apache.spark.sql.functions.{bit_xor, col, count, lit, xxhash64}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.BinaryType
+import org.apache.spark.util.Utils
+
+/**
+ * Benchmark to measure storage-filter pushdown in the vectorized Parquet reader. Every case runs
+ * with `spark.sql.parquet.storageFilterPushdown.enabled` off and on, in two forms:
+ * - the join whose runtime bloom filter is the storage filter, which is the path the feature
+ * takes. The join's shuffle and sort are the same in both arms, and they grow with the rows
+ * that survive, so they dilute what the reader saves or costs;
+ * - a scan of the fact table alone under the same bloom filter, built once outside the timed
+ * body, which shows the reader's own share.
+ *
+ * Every case reads a fact table of about 100 MB in 34 row groups. The cases differ in four things:
+ * - the width of a row. One table has 50,000 rows of a long key and a 2 kB value that does not
+ * compress. The other has 1.4M rows of a long key and eight hashed long values;
+ * - the page size. Each table is written twice, in 8 kB pages and in parquet's default ones,
+ * 1 MB or 20,000 rows, whichever comes first. One surviving row makes the reader read its
+ * whole page, so this decides how much a survivor costs. A wide row fills a small page in a
+ * few values, while narrow ones still put about a thousand in it;
+ * - where the surviving keys fall. Keys from a contiguous range put the survivors in a few
+ * pages, so the reader skips the other row groups whole and the other pages of the row groups
+ * that hold them. Keys selected by their hash put survivors anywhere, so the reader skips
+ * only the pages none of them falls in, and decodes the rest range by range;
+ * - the share of the keys selected, 0.1%, 0.5%, 1% and 5%.
+ * Each section's title gives the row groups and rows per value page its table was written with.
+ * After each case it prints what the storage filter excluded, which does not depend on the
+ * machine.
+ *
+ * To run this benchmark:
+ * {{{
+ * 1. without sbt: bin/spark-submit --class <this class>
+ * --jars <spark core test jar>,<spark catalyst test jar> <spark sql test jar>
+ * 2. build/sbt "sql/Test/runMain <this class>"
+ * 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/Test/runMain <this class>"
+ * Results will be written to "benchmarks/StorageFilterPushdownBenchmark-results.txt".
+ * }}}
+ */
+object StorageFilterPushdownBenchmark extends SqlBasedBenchmark {
+
+ override def getSparkSession: SparkSession = {
+ val conf = new SparkConf()
+ .setAppName(this.getClass.getSimpleName)
+ // Since `spark.master` always exists, overrides this value
+ .set("spark.master", "local[1]")
+ .setIfMissing("spark.driver.memory", "3g")
+ .setIfMissing("spark.executor.memory", "3g")
+ .set(SQLConf.SHUFFLE_PARTITIONS.key, "1")
+ .set("spark.ui.enabled", "false")
+
+ SparkSession.builder().config(conf).getOrCreate()
+ }
+
+ // A fact table: its name and what its section title calls it, how many rows it has, how many of
+ // them a row group holds, and its value columns.
+ private case class Table(
+ name: String,
+ description: String,
+ numRows: Long,
+ rowsPerRowGroup: Int,
+ values: Seq[String])
+
+ // The values are hashed, so they neither compress nor dictionary-encode away, and the bytes a
+ // skipped page saves are real.
+ private val tables = Seq(
+ // 64 hashes of 32 bytes each.
+ Table("wide", "2 kB values", 50000L, 1500, Seq("unhex(concat_ws('', transform(" +
+ "sequence(0, 63), i -> sha2(concat(CAST(id AS STRING), '-', CAST(i AS STRING)), 256)))) " +
+ "AS v0")),
+ Table("narrow", "eight long values", 1400000L, 42000,
+ (0 until 8).map(i => s"xxhash64(id, $i) AS v$i")))
+
+ // The two ways each fact table is written: its layout's name, and the writer options that set
+ // its pages. Parquet checks a page's size only every 100 rows by default, so small pages need
+ // it checked after every row.
+ private val layouts = Seq(
+ "small_pages" -> Map(
+ ParquetOutputFormat.PAGE_SIZE -> "8192",
+ ParquetOutputFormat.MIN_ROW_COUNT_FOR_PAGE_SIZE_CHECK -> "1"),
+ "default_pages" -> Map.empty[String, String])
+
+ // Each share of the keys selected, in thousandths.
+ private val selected = Seq("0.1%" -> 1, "0.5%" -> 5, "1%" -> 10, "5%" -> 50)
+
+ private def prepareTables(dir: File): Unit = tables.foreach { table =>
+ val rows = spark.range(0, table.numRows, 1, 1).selectExpr("id AS k" +: table.values: _*)
+ layouts.foreach { case (layout, options) =>
+ val path = new File(dir, fact(table, layout)).getCanonicalPath
+ rows.write
+ .option(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT, table.rowsPerRowGroup.toString)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .options(options)
+ .parquet(path)
+ spark.read.parquet(path).createOrReplaceTempView(fact(table, layout))
+ }
+
+ def writeDimension(name: String, predicate: String): Unit = {
+ val path = new File(dir, name).getCanonicalPath
+ // `sel_id` exists for the query's filter on the dimension side, which is what makes the
+ // optimizer build a bloom filter from it at all.
+ spark.range(0, table.numRows, 1, 1).where(predicate).selectExpr("id AS k", "id AS sel_id")
+ .write.parquet(path)
+ spark.read.parquet(path).createOrReplaceTempView(name)
+ }
+ selected.foreach { case (label, perMille) =>
+ val start = table.numRows / 2
+ writeDimension(dimension(table, "clustered", label),
+ s"id >= $start AND id < ${start + table.numRows * perMille / 1000}")
+ writeDimension(dimension(table, "scattered", label),
+ s"pmod(xxhash64(id), 1000) < $perMille")
+ }
+ }
+
+ private def fact(table: Table, layout: String): String = s"fact_${table.name}_$layout"
+
+ private def dimension(table: Table, placement: String, label: String): String =
+ s"${table.name}_${placement}_${label.replace("%", "").replace(".", "_")}"
+
+ // How many row groups a fact table has, and how many rows its first value column's pages hold
+ // on average.
+ private def layoutOf(table: Table, layout: String): (Int, Long) = {
+ val file = new Path(spark.table(fact(table, layout)).inputFiles.head)
+ Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(file, spark.sessionState.newHadoopConf()))) { reader =>
+ val rowGroups = reader.getRowGroups.asScala
+ val valuePages = rowGroups.map { rowGroup =>
+ val value = rowGroup.getColumns.asScala.find(_.getPath.toDotString == "v0").get
+ reader.readOffsetIndex(value).getPageCount
+ }.sum
+ (rowGroups.size, table.numRows / valuePages)
+ }
+ }
+
+ private def query(fact: String, dimension: String): String =
+ s"SELECT f.* FROM $fact f JOIN $dimension d ON f.k = d.k WHERE d.sel_id >= 0"
+
+ // The fact table under the bloom filter the join builds from `dimension`, with no join. The
+ // bloom is built once, outside any timed body, the way `InjectRuntimeFilter` builds it for a
+ // creation side with no row count, which these temp views are.
+ private def scanOnly(fact: String, dimension: String): () => DataFrame = {
+ val keyHash = new XxHash64(Seq(UnresolvedAttribute("k")))
+ val bloom = spark.table(dimension)
+ .select(column(new BloomFilterAggregate(keyHash).toAggregateExpression()))
+ .head().getAs[Array[Byte]](0)
+ val mightContain = BloomFilterMightContain(Literal(bloom, BinaryType), keyHash)
+ () => spark.table(fact).where(column(mightContain))
+ }
+
+ private def factScan(plan: SparkPlan): FileSourceScanExec =
+ plan.collectFirst {
+ case scan: FileSourceScanExec if scan.output.exists(_.name == "v0") => scan
+ }.getOrElse(throw new IllegalStateException(s"No fact table scan in $plan"))
+
+ private def withPushdown[T](enabled: Boolean)(f: => T): T =
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> enabled.toString)(f)
+
+ // Asserts that `df` plans its bloom filter, and hands it to the fact scan exactly when
+ // pushdown is on, so neither arm can silently measure the other. Returns that scan.
+ private def checkedFactScan(df: DataFrame, enabled: Boolean): FileSourceScanExec = {
+ val plan = df.queryExecution.executedPlan
+ val hasBloom = plan.exists { node =>
+ node.expressions.exists(_.exists(_.isInstanceOf[BloomFilterMightContain]))
+ }
+ assert(hasBloom, s"The plan must have a bloom filter:\n${df.queryExecution}")
+ val scan = factScan(plan)
+ assert(scan.storageFilters.nonEmpty == enabled,
+ s"The fact scan must have a storage filter exactly when pushdown is on:\n$scan")
+ scan
+ }
+
+ // Runs `df` with pushdown off and on. Then, untimed, asserts that both arms return the same
+ // rows, counted and fingerprinted over every column, so a faster arm cannot be a wrong one.
+ // Returns the benchmark that timed it.
+ private def runOffAndOn(title: String, numRows: Long, df: () => DataFrame): Benchmark = {
+ val benchmark = new Benchmark(title, numRows, minNumIters = 3, output = output)
+ Seq(false, true).foreach { enabled =>
+ withPushdown(enabled)(checkedFactScan(df(), enabled))
+ benchmark.addCase(s"Storage filter pushdown ${if (enabled) "on" else "off"}") { _ =>
+ withPushdown(enabled)(df().noop())
+ }
+ }
+ benchmark.run()
+ def fingerprint(enabled: Boolean) = withPushdown(enabled) {
+ val rows = df()
+ val fingerprinted =
+ rows.select(count(lit(1)), bit_xor(xxhash64(rows.columns.map(col).toIndexedSeq: _*)))
+ checkedFactScan(fingerprinted, enabled)
+ fingerprinted.head()
+ }
+ val (off, on) = (fingerprint(enabled = false), fingerprint(enabled = true))
+ assert(off == on, s"$title: both arms must return the same rows; got $off and $on")
Review Comment:
Nit: this check has not run yet. All three results files come from the benchmark workflow runs on c98bdce on 2026-10-03, before it was added, and CI only compiles benchmarks, while the description presents it next to "The results files come from the GitHub Actions benchmark workflow". Could the workflow be re-run once, JDK 17 alone being about 19 minutes, so the assertion has executed at least once? Separately, `fingerprinted.collect().head` would run the plan `checkedFactScan` just checked, where `head()` plans a `limit(1)` anew, though `EliminateLimits` makes them the same shape.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3708 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // The native address of an off-heap vector's values, 0 once it is freed. Only for a fixed-width
+ // vector. A string or binary one keeps its bytes in a child and reports 0 even while live.
+ private def address(vector: Any): Long =
Review Comment:
Nit: this splits `readLongs` from `asTask`, which round 10 moved next to the format-reader helpers, while every caller of `address` uses the direct-reader helpers further down. Could it go after the log-message constants (~L264)?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1032 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A fixed-width key needs no
+ * factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. It is part of `keyFixedBytesPerRow`, counted
+ * twice, for its data byte and its null byte.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // decoders of their last page.
+ closeDataPages();
+ releaseRowGroupReaders();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (reportedMessages.add(OFFSET_INDEX_UNREADABLE)) {
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn(OFFSET_INDEX_UNREADABLE, MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_DECODING, true);
+ // The failed decode may have grown the scratch vectors, and phase 2 reads the same pages
+ // into vectors of its own. So capacity-sized ones replace them, and beside a plain read's
+ // own vectors the reader then holds one capacity-sized batch of key vectors.
+ closeAll(keyScratchVectors);
+ allocateKeyScratch();
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = predicate.eval(keyScratchBatch.getRow(r));
+ } catch (Throwable t) {
+ // Fall back on anything but a task kill or a fatal error. The predicate runs here without
+ // the conjuncts before it in the plan, so it can throw on a row a plain scan never
+ // evaluates it on (see `ParquetStorageFilter.isSupportedStorageFilter`). Giving the
+ // filter up returns every row the pushed filter kept, and the post-scan Filter decides
+ // them in its own order, raising the same error for a row it does evaluate.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK exception
+ // (`timestamp_seconds` of a decimal) or an `AssertionError`, and a Hive UDF throws a
+ // checked SparkException. Rethrowing would fail a query a plain read answers, or drop
+ // the rest of the file silently under `ignoreCorruptFiles`.
+ //
+ // A fatal error goes out even on a row an earlier conjunct of the plan would have
+ // rejected, so the reader never runs on after one the executor would take as fatal.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_EVALUATION, false);
+ return null;
+ }
+ if (!survives) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ if (currentKeyAccumulators == null) allocateKeyAccumulators();
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // The buffer is this reader's own and optional, so failing to fill it gives splicing up
+ // rather than the filter, and the memory goes back at once. What fails here:
+ // - a survivor vector that cannot grow;
+ // - a key value the predicate did not read on this row, such as `b` in
+ // `coalesce(a, b)`, since the copy decodes every key;
+ // - a bug of ours, which only the WARN then shows.
+ // Phase 2 reads the key columns the way a plain read does, so a value that cannot be
+ // decoded fails the read only where a plain read fails.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together.
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, charged per row as survivors
+ * arrive. A later set grows the way the first one does rather than being sized from the set
+ * before it. Sizing would save a long key's byte child its regrowth copies. But the memory it
Review Comment:
Nit: this describes the removed sizing in the past tense ("charging it made ...", "A larger cap could then ..."), which reads as history of code the merged class never had, and "a long key" reads as a `LongType` key here, as the suite uses it, which has no byte child. Could it say, e.g., "Sizing a set from the one before it would save a variable-length key with long values its regrowth copies, but the memory taken up front is a prediction, and charging it would make a row group's charge depend on the cap, so a larger cap could give splicing up where a smaller one keeps it"? The second sentence of the `DEFAULT_CHILD_BYTES_PER_ROW` javadoc (L129-130) also now only repeats the comment at L461-463.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3708 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // The native address of an off-heap vector's values, 0 once it is freed. Only for a fixed-width
+ // vector. A string or binary one keeps its bytes in a child and reports 0 even while live.
+ private def address(vector: Any): Long =
+ vector.asInstanceOf[OffHeapColumnVector].valuesNativeAddress()
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t @ (_: VirtualMachineError | _: LinkageError) =>
+ fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`, and returns the rows.
+ private def readLaterSet(path: String, cap: Long): Seq[(String, String)] = {
Review Comment:
Nit, optional: with its other caller gone this round, `readLaterSet` has one caller (L1215), and `laterSetFilter` serves only that test, so both could be local to it.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1032 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A fixed-width key needs no
+ * factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. It is part of `keyFixedBytesPerRow`, counted
+ * twice, for its data byte and its null byte.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // decoders of their last page.
+ closeDataPages();
+ releaseRowGroupReaders();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (reportedMessages.add(OFFSET_INDEX_UNREADABLE)) {
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn(OFFSET_INDEX_UNREADABLE, MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_DECODING, true);
+ // The failed decode may have grown the scratch vectors, and phase 2 reads the same pages
+ // into vectors of its own. So capacity-sized ones replace them, and beside a plain read's
+ // own vectors the reader then holds one capacity-sized batch of key vectors.
+ closeAll(keyScratchVectors);
+ allocateKeyScratch();
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = predicate.eval(keyScratchBatch.getRow(r));
+ } catch (Throwable t) {
+ // Fall back on anything but a task kill or a fatal error. The predicate runs here without
+ // the conjuncts before it in the plan, so it can throw on a row a plain scan never
+ // evaluates it on (see `ParquetStorageFilter.isSupportedStorageFilter`). Giving the
+ // filter up returns every row the pushed filter kept, and the post-scan Filter decides
+ // them in its own order, raising the same error for a row it does evaluate.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK exception
+ // (`timestamp_seconds` of a decimal) or an `AssertionError`, and a Hive UDF throws a
+ // checked SparkException. Rethrowing would fail a query a plain read answers, or drop
+ // the rest of the file silently under `ignoreCorruptFiles`.
+ //
+ // A fatal error goes out even on a row an earlier conjunct of the plan would have
+ // rejected, so the reader never runs on after one the executor would take as fatal.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_EVALUATION, false);
+ return null;
+ }
+ if (!survives) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ if (currentKeyAccumulators == null) allocateKeyAccumulators();
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // The buffer is this reader's own and optional, so failing to fill it gives splicing up
+ // rather than the filter, and the memory goes back at once. What fails here:
+ // - a survivor vector that cannot grow;
+ // - a key value the predicate did not read on this row, such as `b` in
+ // `coalesce(a, b)`, since the copy decodes every key;
+ // - a bug of ours, which only the WARN then shows.
+ // Phase 2 reads the key columns the way a plain read does, so a value that cannot be
+ // decoded fails the read only where a plain read fails.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together.
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, charged per row as survivors
+ * arrive. A later set grows the way the first one does rather than being sized from the set
+ * before it. Sizing would save a long key's byte child its regrowth copies. But the memory it
+ * takes up front is a prediction, and charging it made what a row group is charged depend on the
+ * cap. A larger cap could then give splicing up where a smaller one kept it.
+ */
+ private void allocateKeyAccumulators() {
+ // Assigned before the loop, so an allocation failure part way through leaves the vectors
+ // allocated so far reachable for `abandonSplicing`.
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ currentKeyAccumulators[i] = newKeyVector(i);
+ }
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, which the caller has
+ * allocated, and hands them to the queue once they are full. Returns the bytes the row added,
+ * which the caller weighs against its budget.
+ */
+ private long appendSurvivorRowToAccumulators(int srcRow) {
+ long charged = 0L;
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final KeyColumn[] keys = keyColumns;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ KeyColumn key = keys[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ key.copier().copy(dst, dstRow, src, srcRow);
+ // Measured on the destination, because a dictionary-encoded source has no length of its
+ // own, its values being read through the dictionary.
+ if (key.variableLength()) {
+ charged += VARIABLE_LENGTH_BYTES_FACTOR * dst.getArrayLength(dstRow);
Review Comment:
Minor: this line is pinned only for one plain-encoded variable-length key. The exact-charge tests (103, 32142, 24222 and 166) read long keys or files written with `writeSingleParquetFile`'s default `dictionary = false`, and the dictionary-encoded string and binary arms of the type matrix read under the default `Long.MaxValue` cap. So measuring the source instead, e.g. to check the cap before the copy, would pass all 122 tests, though phase 1 decodes a dictionary-encoded key lazily and writes no lengths for it, leaving the cap to bound only the 17-byte fixed part. Two string keys are not charged together anywhere either, so `charged =` instead of `+=` would pass too. Could a dictionary-encoded variant of the 32142 fixture go through `assertSpliceCharge`, or a two-string-key variant of the 166 test, here or in a follow-up?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3708 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // The native address of an off-heap vector's values, 0 once it is freed. Only for a fixed-width
+ // vector. A string or binary one keeps its bytes in a child and reports 0 even while live.
+ private def address(vector: Any): Long =
+ vector.asInstanceOf[OffHeapColumnVector].valuesNativeAddress()
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t @ (_: VirtualMachineError | _: LinkageError) =>
+ fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`, and returns the rows.
+ private def readLaterSet(path: String, cap: Long): Seq[(String, String)] = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString),
+ capacity = 4)
+ reader.close()
+ rows
+ }
+
+ test("a later accumulator set of a variable-length key is charged per value") {
+ // Six survivors over a capacity of 4 fill one set of four 1000-byte keys and a second of two
+ // 2000-byte ones. Every set grows as its values arrive, the way the first one does. Each value
+ // is charged four times its bytes plus its row's fixed part of 17, and the one range adds 40.
+ // So the row group is charged 4 * (4000 + 17) + 2 * (8000 + 17) + 40 = 32142 bytes. A cap of
+ // 32142 keeps splicing and one of 32141 gives it up, which a missing or a double charge would
+ // each get wrong. And the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ assertSpliceCharge(32142L, rows = 6) { cap =>
+ readUnderCap(path, laterSetSchema, laterSetFilter, cap, batchSize = 4)
+ }
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val rows = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ }
+ }
+ }
+
+ test("what a row group of scattered survivors is charged does not depend on the cap") {
+ // Keys 9 to 12 fill the first set over a capacity of 4, and keys 14 and 17 come scattered, each
+ // in a range of its own. Every set grows as values arrive and is charged per value, so the row
+ // group's charge is one number: the first set's 16068, three ranges of 40 and the second
+ // set's 8034, 24222 in all. A cap of it splices and one byte under it gives splicing up.
+ // Larger caps splice too, which a few of them check, among them 32216 to 32221. Those gave
+ // splicing up when a later set was sized from the one before it with no room for its ranges.
+ // The second set is not sized at all, so its byte child grows to 2000 bytes for its two
+ // 1000-byte values, where sizing it from the first set would have reserved 8000.
+ withTempDir { dir =>
+ val (path, _) = writeLaterSetFile(dir, lastTwoBytes = 1000)
+ val prefix = Substring(BoundReference(0, StringType, nullable = true), Literal(1), Literal(4))
+ val keys = Seq("0009", "0010", "0011", "0012", "0014", "0017").map(Literal(_))
+ def readUnder(cap: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, Seq(In(prefix, keys)), cap, batchSize = 4)
+ assertSpliceCharge(24222L, rows = 6)(readUnder)
+ val (_, uncapped) = readUnder("64MB")
+ Seq(32215L, 32216L, 32221L, 32222L).foreach { cap =>
+ val (rows, bytes) = readUnder(s"${cap}b")
+ assert(rows == 6 && bytes == uncapped,
+ s"a cap of $cap must keep the six rows and splicing; got $rows rows")
+ }
+ val (children, reader) = readAllWith(path, Seq("k", "v"),
+ createFilter(Seq(In(prefix, keys)), laterSetSchema),
+ (b, _) => ParquetTestAccess.byteChildCapacity(
+ b.column(0).asInstanceOf[WritableColumnVector]),
+ capacity = 4)
+ reader.close()
+ assert(children.drop(4).distinct == Seq(2000),
+ s"the second set's byte child must grow as values arrive; got $children")
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ assertSpliceCharge(166L, rows = 7) { cap =>
+ readUnderCap(path, schema, storageFilters, cap, batchSize = 16)
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warnings) = withLogged(partialReadWarning) {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(warnings == 0,
+ s"the footer check must get there before the exception path; got $warnings")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warnings) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(warnings == 0, "the footer check must decline the row group rather than the read " +
+ s"discovering it; got $warnings")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize = 64L * 1024 * 1024,
+ options = Map(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT -> "100"))
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ // The rows a reader hands back before it throws, what it throws, and, for the filtering
+ // reader, whether its scratch vectors were replaced on the way and the old ones freed. Both
+ // read off heap, where an old scratch vector left open would leak native memory.
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable, Boolean) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(true, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(true, 16, storageFilter)
+ }
+ def scratch: Seq[WritableColumnVector] = reader match {
+ case filtering: LateMaterializationParquetRecordReader =>
+ ParquetTestAccess.keyScratchVectors(filtering)
+ case _ => Nil
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val scratchBefore = scratch
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ val replacedAndFreed = scratch.zip(scratchBefore).exists { case (now, before) =>
+ (now ne before) && address(before) == 0
+ }
+ (read.toSeq, error, replacedAndFreed)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError, _) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError, scratchReplaced) =
+ readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ // A failed decode can leave the scratch vectors grown. Phase 2 reads the same pages into
+ // vectors of its own, so the reader replaces them with capacity-sized ones rather than keep
+ // the grown ones for the rest of the split.
+ assert(scratchReplaced,
+ "the decoding fallback must replace the scratch vectors and free the old ones")
+
+ // A kill that shows only at the decoding catch, after the per-chunk check has passed, is
+ // seen there. Without that catch's own check the row group would fall back and phase 2
+ // would fail on the page instead.
+ val killedAtTheCatch = spy(TaskContext.empty())
+ doAnswer((_: InvocationOnMock) => Boolean.box(Thread.currentThread().getStackTrace
+ .exists(_.getMethodName.startsWith("rethrowIfMustPropagate"))))
+ .when(killedAtTheCatch).isInterrupted()
+ val (_, killed, _) = TaskContext.withTaskContext(killedAtTheCatch) {
+ readUntilFailure(keyAtLeastFilter(500L))
+ }
+ assert(killed.isInstanceOf[SparkException] && killed.getMessage.contains("task kill") &&
+ killed.getCause != null && killed.getCause.getClass == plainError.getClass,
+ s"the kill must go out from the decoding catch, around the decoding error; got $killed")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+
+ test("a key vector that cannot grow, or a codec's InternalError, gives the row group up") {
+ // Both are fatal-looking errors that decoding the key pages can raise, and the reader falls
+ // back on both. A scratch vector that cannot grow is phase 1's own, and phase 2 then reads
+ // what a plain read reads. An `InternalError` from decoding is how a codec like hadoop-lzo
+ // reports a corrupt page, which the plain read of the same pages meets again. Raised from the
+ // filter instead, the first goes out as its OutOfMemoryError and the second in an
+ // `UnknownError`. Off heap, the scratch vectors the failed decode used must be freed, and the
+ // second row group's phase 1 runs over the ones that replace them.
+ withTempDir { dir =>
+ val rows = (0L until 40L).map(i => (i, i.toString))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "20"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(20L, 20L),
+ "the fixture must be two row groups of 20 rows")
+ Seq(
+ new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve")),
+ new InternalError("codec")).foreach { error =>
+ var failedScratch: Seq[WritableColumnVector] = Nil
+ // Only the first row group's key reader fails.
+ val failing: KeyReaderHook.OnKeyReader = (reader, keyReader) => {
+ val scratch = ParquetTestAccess.keyScratchVectors(
+ reader.asInstanceOf[LateMaterializationParquetRecordReader])
+ if (failedScratch.nonEmpty) {
+ // Checked before phase 1 writes into them, since a freed one would crash the JVM.
+ assert(scratch.forall(v => !failedScratch.exists(_ eq v) && address(v) != 0),
+ "the second row group must decode into new scratch vectors")
+ keyReader
+ } else {
+ failedScratch = scratch
+ val failingReader = spy(keyReader)
+ doThrow(error).when(failingReader).readBatch(anyInt(), any(), any(), any())
+ failingReader
+ }
+ }
+ val ((result, reader), warnings) = try {
+ withLogged(decodingGivenUp) {
+ readAll(path, keyAtLeastFilter(30L), useOffHeap = true, onKeyReader = failing)
+ }
+ } catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case e: VirtualMachineError => fail(s"the reader must fall back on $error; got $e", e)
+ }
+ try {
+ assert(result == rows.take(20) ++ rows.drop(30),
+ s"the first row group must come back whole and the second filtered on $error; got " +
+ s"${result.map(_._1)}")
+ assert(warnings == 1, s"the decoding fallback must report $error; got $warnings")
+ assert(failedScratch.forall(v => address(v) == 0),
+ s"the scratch vectors of the failed decode must be freed on $error")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("a key value the filter does not read and cannot be decoded gives splicing up") {
+ // The filter need not read every key on every row, as `coalesce(a, b)` reads `b` only where
+ // `a` is null. The copy into the survivor buffer reads every key, so it can meet a value no
+ // plain read decodes. A dictionary-encoded column decodes a value only when it is read, so a
+ // dictionary that holds fewer values than its ids point at is such a value. Copying it gives
+ // splicing up for the row group, and phase 2 reads the key columns the way a plain read does.
+ // So the read answers where a plain one does, and the filter still applies.
+ withTempDir { dir =>
+ // Two copies of each `b`, so parquet keeps its dictionary. Cut to 64 values, it leaves `b`
+ // undecodable from row 128 on. `a` is null only before that, so the filter never reads a
+ // damaged `b`.
+ val rows = (0 until 200).map { i =>
+ (if (i < 100 && i % 5 == 0) null else s"a_$i", s"b_${i / 2}", i)
+ }
+ val path = writeSingleParquetFile(dir, rows.toDF("a", "b", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE, dictionary = true)
+ assert(encodingsOf(path, "b").exists(_.usesDictionary),
+ s"the fixture's whole point; got ${encodingsOf(path, "b")}")
+ shrinkDictionary(path, column = "b", entries = 64)
+ val columns = Seq("a", "b", "v")
+ intercept[ArrayIndexOutOfBoundsException] {
+ readAllWith(path, columns, null, (b, i) => b.column(1).getUTF8String(i).toString)._2
+ .close()
+ }
+
+ val schema = StructType(Seq(
+ StructField("a", StringType), StructField("b", StringType), StructField("v", IntegerType)))
+ val coalesced = Coalesce(Seq(BoundReference(0, StringType, nullable = true),
+ BoundReference(1, StringType, nullable = true)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(coalesced, Literal.create("a_15", StringType))), schema)
+ val ((result, reader), givenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, columns, filter, (b, i) => b.column(2).getInt(i), capacity = 16)
+ }
+ try {
+ // Keeps rows 150-199 among others, whose `b` is damaged.
+ val expected = rows.filter { case (a, b, _) => Option(a).getOrElse(b) >= "a_15" }.map(_._3)
+ assert(expected.contains(150), "the filter must keep a row whose `b` is damaged")
+ assert(result == expected, s"got ${result.size} rows, expected ${expected.size}")
+ assert(givenUp == 1,
+ "the copy must have failed and given splicing up once, or this proves nothing; " +
+ s"logged $givenUp")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry reads the damaged row group whole over the pushed data filter's ranges, and only
+ // that one, since parquet keeps a column index store per block. What it must not give up is
+ // the pushed data filter's own page pruning, which a plain read of the same query keeps, nor
+ // the other row groups' narrowing. Dropping either would read more than it needs to.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warnings) = withLogged(partialReadWarning)(read(evenKeys))
+ assert(warnings == 1,
+ "the read must have discovered the damaged index, or this test proves nothing; got " +
+ warnings)
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ // The row groups around the damaged one keep the storage filter's narrowing too, so they
+ // return even keys alone.
+ val undamaged = filtered.filter(k => k <= 100L || k > 200L)
+ assert(undamaged.nonEmpty && undamaged.forall(_ % 2 == 0),
+ s"the other row groups must still be narrowed; got ${undamaged.filter(_ % 2 != 0)}")
+
+ // The WARN is reported once per split, however many row groups it costs.
+ val twiceDamaged = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(twiceDamaged, blockIdx = 0, columnIdx = 0)
+ breakOffsetIndex(twiceDamaged, blockIdx = 1, columnIdx = 0)
+ val (_, twiceWarned) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, evenKeys), twiceDamaged, ordinal = 0)
+ }
+ assert(twiceWarned == 1, s"two damaged row groups must be reported once; got $twiceWarned")
+ }
+ }
+
+ test("a kill that lands before an unreadable offset index is reported as a kill") {
+ // Parquet turns any IOException reading an offset index into `MissingOffsetIndexException`,
+ // a kill's interrupt included, so the catch asks for a kill before it takes a missing index.
+ // Here the kill lands while the last row group's one chunk is evaluated, after the chunk's own
+ // check, and with no row group after it. So only that catch can see it.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 2, columnIdx = 0)
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val killsAt202 = EqualTo(Remainder(
+ OnKey(BoundReference(0, LongType, nullable = false), at = 202L, killsTask = true),
+ Literal(2L)), Literal(0L))
+ val (thrown, warnings) = withLogged(partialReadWarning) {
+ asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ readLongs(formatReader(schema, pushed, Seq(killsAt202)), path, ordinal = 0)
+ }
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException], s"got $thrown")
+ assert(warnings == 0,
+ s"the kill must not be logged as an unreadable offset index; got $warnings")
+ }
+ }
+
+ test("a v2 file's DELTA_BYTE_ARRAY decimal column is read over scattered survivors") {
+ // Phase 2 skips rows inside a page whenever survivors scatter, where a plain read skips only
+ // when a pushed filter cuts a page. The parquet v2 writer stores a FIXED_LEN_BYTE_ARRAY
+ // decimal without a dictionary as DELTA_BYTE_ARRAY, whose reader could not skip until
+ // SPARK-59828, so this read failed where a plain read of the same file did not.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 1000).selectExpr("id AS k", "CAST(id * 1.25 AS DECIMAL(30, 2)) AS d"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.WRITER_VERSION -> "PARQUET_2_0"))
+ assert(encodingsOf(path, "d").contains(Encoding.DELTA_BYTE_ARRAY),
+ s"the fixture's whole point; got ${encodingsOf(path, "d")}")
+ val everyThirdRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(3L)), Literal(0L))
+ val schema = kvSchema(valueType = DecimalType(30, 2))
+ val (result, reader) = readAllWith(path, Seq("k", "d"),
+ createFilter(Seq(everyThirdRow), schema),
+ (b, i) => (b.column(0).getLong(i), b.column(1).getDecimal(i, 30, 2).toJavaBigDecimal))
+ try {
+ val expected = (0L until 1000L by 3L)
+ .map(k => (k, new java.math.BigDecimal(k).multiply(new java.math.BigDecimal("1.25"))
+ .setScale(2)))
+ assert(result == expected, s"got ${result.take(5)} (${result.size} rows)")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the survivor queue and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ val fileSchema = kvSchema()
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAll(path, filter, capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // Each survivor costs 9 bytes of long key and 40 of its own range. The sum passes the cap of
+ // 1024 at survivor 21, which releases the buffer, and the ranges alone at survivor 26, which
+ // gives the filter up. The capacity decides what the buffer holds when it is released. At
+ // 4096 that is one partly filled set of accumulators, and at 16 a full set in the queue as
+ // well. Both have to end with the queue empty and the filter given up. Keeping one without
+ // the other would leave phase 2 reading non-key columns while emit spliced keys from a queue
+ // holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAll(path, filter)
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSeq == Seq("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does, since the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers conjuncts under the relation's own column names") {
+ // The analyzer spells a reference the way the query did, so under case-insensitive resolution
+ // a bloom can name `_TMP_METADATA_ROW_INDEX` while the relation, and the schema the reader
+ // requests, call the column `_tmp_metadata_row_index`. The planner normalizes what it offers to
+ // the relation's names, so the format's gate sees the name the reader will see and declines.
+ // Without that the gate passed it, `ParquetStorageFilter.create` rejected it on the driver, and
+ // a query that runs with the conf off failed with it on.
+ withTempDir { dir =>
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", s"id AS $rowIndexName", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", rowIndexName, "v")
+ .queryExecution.optimizedPlan
+ def column(name: String): Attribute =
+ relation.output.find(_.name == name).getOrElse(fail(s"no $name in the relation output"))
+
+ def offered(key: Attribute): Seq[Expression] = {
+ val logical = LogicalFilter(
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ scanOf(physical).storageFilters
+ }
+
+ // As a control, an ordinary key spelled in another case is offered, under its own name.
+ val upperK = offered(column("k").withName("K"))
+ assert(upperK.size == 1, s"a bloom over `K` must reach the scan; got $upperK")
+ assert(upperK.head.references.map(_.name).toSet == Set("k"),
+ s"and it must carry the relation's name for the column; got ${upperK.head}")
+
+ val upperRowIndex =
+ offered(column(rowIndexName).withName(rowIndexName.toUpperCase(Locale.ROOT)))
+ assert(upperRowIndex.isEmpty,
+ s"a bloom over the row-index column must not be offered in any case; got $upperRowIndex")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers a conjunct that appears twice only once") {
+ // Spelled in two cases, a bloom over `k` normalizes to one conjunct, which the scan then takes
+ // once, as `afterScanFilters` does.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+ def bloomOn(key: Attribute): Expression =
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key)))
+ val logical = LogicalFilter(And(bloomOn(k), bloomOn(k.withName("K"))), relation)
Review Comment:
Nit: this repeats the planning scaffolding of the two tests above (L1928-1935, L1979-1985) and writes its own file, and the `k`/`K` spelling adds nothing, since `ExpressionSet` canonicalizes attribute names away, so the pair collapses with or without normalization. Could the L1960 test's `offered` take `keys: Attribute*` and AND their blooms, with `assert(offered(k, k).size == 1)` there? It still fails without the `ExpressionSet` at `FileSourceStrategy.scala:202`. If it stays separate, the comment could say spelled twice rather than credit normalization.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3708 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // The native address of an off-heap vector's values, 0 once it is freed. Only for a fixed-width
+ // vector. A string or binary one keeps its bytes in a child and reports 0 even while live.
+ private def address(vector: Any): Long =
+ vector.asInstanceOf[OffHeapColumnVector].valuesNativeAddress()
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t @ (_: VirtualMachineError | _: LinkageError) =>
+ fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`, and returns the rows.
+ private def readLaterSet(path: String, cap: Long): Seq[(String, String)] = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString),
+ capacity = 4)
+ reader.close()
+ rows
+ }
+
+ test("a later accumulator set of a variable-length key is charged per value") {
+ // Six survivors over a capacity of 4 fill one set of four 1000-byte keys and a second of two
+ // 2000-byte ones. Every set grows as its values arrive, the way the first one does. Each value
+ // is charged four times its bytes plus its row's fixed part of 17, and the one range adds 40.
+ // So the row group is charged 4 * (4000 + 17) + 2 * (8000 + 17) + 40 = 32142 bytes. A cap of
+ // 32142 keeps splicing and one of 32141 gives it up, which a missing or a double charge would
+ // each get wrong. And the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ assertSpliceCharge(32142L, rows = 6) { cap =>
+ readUnderCap(path, laterSetSchema, laterSetFilter, cap, batchSize = 4)
+ }
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val rows = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ }
+ }
+ }
+
+ test("what a row group of scattered survivors is charged does not depend on the cap") {
+ // Keys 9 to 12 fill the first set over a capacity of 4, and keys 14 and 17 come scattered, each
+ // in a range of its own. Every set grows as values arrive and is charged per value, so the row
+ // group's charge is one number: the first set's 16068, three ranges of 40 and the second
+ // set's 8034, 24222 in all. A cap of it splices and one byte under it gives splicing up.
+ // Larger caps splice too, which a few of them check, among them 32216 to 32221. Those gave
+ // splicing up when a later set was sized from the one before it with no room for its ranges.
+ // The second set is not sized at all, so its byte child grows to 2000 bytes for its two
+ // 1000-byte values, where sizing it from the first set would have reserved 8000.
+ withTempDir { dir =>
+ val (path, _) = writeLaterSetFile(dir, lastTwoBytes = 1000)
+ val prefix = Substring(BoundReference(0, StringType, nullable = true), Literal(1), Literal(4))
+ val keys = Seq("0009", "0010", "0011", "0012", "0014", "0017").map(Literal(_))
+ def readUnder(cap: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, Seq(In(prefix, keys)), cap, batchSize = 4)
+ assertSpliceCharge(24222L, rows = 6)(readUnder)
+ val (_, uncapped) = readUnder("64MB")
+ Seq(32215L, 32216L, 32221L, 32222L).foreach { cap =>
+ val (rows, bytes) = readUnder(s"${cap}b")
+ assert(rows == 6 && bytes == uncapped,
+ s"a cap of $cap must keep the six rows and splicing; got $rows rows")
+ }
+ val (children, reader) = readAllWith(path, Seq("k", "v"),
+ createFilter(Seq(In(prefix, keys)), laterSetSchema),
+ (b, _) => ParquetTestAccess.byteChildCapacity(
+ b.column(0).asInstanceOf[WritableColumnVector]),
+ capacity = 4)
+ reader.close()
+ assert(children.drop(4).distinct == Seq(2000),
+ s"the second set's byte child must grow as values arrive; got $children")
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ assertSpliceCharge(166L, rows = 7) { cap =>
+ readUnderCap(path, schema, storageFilters, cap, batchSize = 16)
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warnings) = withLogged(partialReadWarning) {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(warnings == 0,
+ s"the footer check must get there before the exception path; got $warnings")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warnings) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(warnings == 0, "the footer check must decline the row group rather than the read " +
+ s"discovering it; got $warnings")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize = 64L * 1024 * 1024,
+ options = Map(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT -> "100"))
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ // The rows a reader hands back before it throws, what it throws, and, for the filtering
+ // reader, whether its scratch vectors were replaced on the way and the old ones freed. Both
+ // read off heap, where an old scratch vector left open would leak native memory.
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable, Boolean) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(true, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(true, 16, storageFilter)
+ }
+ def scratch: Seq[WritableColumnVector] = reader match {
+ case filtering: LateMaterializationParquetRecordReader =>
+ ParquetTestAccess.keyScratchVectors(filtering)
+ case _ => Nil
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val scratchBefore = scratch
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ val replacedAndFreed = scratch.zip(scratchBefore).exists { case (now, before) =>
+ (now ne before) && address(before) == 0
+ }
+ (read.toSeq, error, replacedAndFreed)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError, _) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError, scratchReplaced) =
+ readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ // A failed decode can leave the scratch vectors grown. Phase 2 reads the same pages into
+ // vectors of its own, so the reader replaces them with capacity-sized ones rather than keep
+ // the grown ones for the rest of the split.
+ assert(scratchReplaced,
+ "the decoding fallback must replace the scratch vectors and free the old ones")
+
+ // A kill that shows only at the decoding catch, after the per-chunk check has passed, is
+ // seen there. Without that catch's own check the row group would fall back and phase 2
+ // would fail on the page instead.
+ val killedAtTheCatch = spy(TaskContext.empty())
+ doAnswer((_: InvocationOnMock) => Boolean.box(Thread.currentThread().getStackTrace
+ .exists(_.getMethodName.startsWith("rethrowIfMustPropagate"))))
+ .when(killedAtTheCatch).isInterrupted()
+ val (_, killed, _) = TaskContext.withTaskContext(killedAtTheCatch) {
+ readUntilFailure(keyAtLeastFilter(500L))
+ }
+ assert(killed.isInstanceOf[SparkException] && killed.getMessage.contains("task kill") &&
+ killed.getCause != null && killed.getCause.getClass == plainError.getClass,
+ s"the kill must go out from the decoding catch, around the decoding error; got $killed")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+
+ test("a key vector that cannot grow, or a codec's InternalError, gives the row group up") {
+ // Both are fatal-looking errors that decoding the key pages can raise, and the reader falls
+ // back on both. A scratch vector that cannot grow is phase 1's own, and phase 2 then reads
+ // what a plain read reads. An `InternalError` from decoding is how a codec like hadoop-lzo
+ // reports a corrupt page, which the plain read of the same pages meets again. Raised from the
+ // filter instead, the first goes out as its OutOfMemoryError and the second in an
+ // `UnknownError`. Off heap, the scratch vectors the failed decode used must be freed, and the
+ // second row group's phase 1 runs over the ones that replace them.
+ withTempDir { dir =>
+ val rows = (0L until 40L).map(i => (i, i.toString))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "20"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(20L, 20L),
+ "the fixture must be two row groups of 20 rows")
+ Seq(
+ new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve")),
+ new InternalError("codec")).foreach { error =>
+ var failedScratch: Seq[WritableColumnVector] = Nil
+ // Only the first row group's key reader fails.
+ val failing: KeyReaderHook.OnKeyReader = (reader, keyReader) => {
+ val scratch = ParquetTestAccess.keyScratchVectors(
+ reader.asInstanceOf[LateMaterializationParquetRecordReader])
+ if (failedScratch.nonEmpty) {
+ // Checked before phase 1 writes into them, since a freed one would crash the JVM.
+ assert(scratch.forall(v => !failedScratch.exists(_ eq v) && address(v) != 0),
+ "the second row group must decode into new scratch vectors")
+ keyReader
+ } else {
+ failedScratch = scratch
+ val failingReader = spy(keyReader)
+ doThrow(error).when(failingReader).readBatch(anyInt(), any(), any(), any())
+ failingReader
+ }
+ }
+ val ((result, reader), warnings) = try {
+ withLogged(decodingGivenUp) {
+ readAll(path, keyAtLeastFilter(30L), useOffHeap = true, onKeyReader = failing)
+ }
+ } catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case e: VirtualMachineError => fail(s"the reader must fall back on $error; got $e", e)
+ }
+ try {
+ assert(result == rows.take(20) ++ rows.drop(30),
+ s"the first row group must come back whole and the second filtered on $error; got " +
+ s"${result.map(_._1)}")
+ assert(warnings == 1, s"the decoding fallback must report $error; got $warnings")
+ assert(failedScratch.forall(v => address(v) == 0),
+ s"the scratch vectors of the failed decode must be freed on $error")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("a key value the filter does not read and cannot be decoded gives splicing up") {
+ // The filter need not read every key on every row, as `coalesce(a, b)` reads `b` only where
+ // `a` is null. The copy into the survivor buffer reads every key, so it can meet a value no
+ // plain read decodes. A dictionary-encoded column decodes a value only when it is read, so a
+ // dictionary that holds fewer values than its ids point at is such a value. Copying it gives
+ // splicing up for the row group, and phase 2 reads the key columns the way a plain read does.
+ // So the read answers where a plain one does, and the filter still applies.
+ withTempDir { dir =>
+ // Two copies of each `b`, so parquet keeps its dictionary. Cut to 64 values, it leaves `b`
+ // undecodable from row 128 on. `a` is null only before that, so the filter never reads a
+ // damaged `b`.
+ val rows = (0 until 200).map { i =>
+ (if (i < 100 && i % 5 == 0) null else s"a_$i", s"b_${i / 2}", i)
+ }
+ val path = writeSingleParquetFile(dir, rows.toDF("a", "b", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE, dictionary = true)
+ assert(encodingsOf(path, "b").exists(_.usesDictionary),
+ s"the fixture's whole point; got ${encodingsOf(path, "b")}")
+ shrinkDictionary(path, column = "b", entries = 64)
+ val columns = Seq("a", "b", "v")
+ intercept[ArrayIndexOutOfBoundsException] {
+ readAllWith(path, columns, null, (b, i) => b.column(1).getUTF8String(i).toString)._2
+ .close()
+ }
+
+ val schema = StructType(Seq(
+ StructField("a", StringType), StructField("b", StringType), StructField("v", IntegerType)))
+ val coalesced = Coalesce(Seq(BoundReference(0, StringType, nullable = true),
+ BoundReference(1, StringType, nullable = true)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(coalesced, Literal.create("a_15", StringType))), schema)
+ val ((result, reader), givenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, columns, filter, (b, i) => b.column(2).getInt(i), capacity = 16)
+ }
+ try {
+ // Keeps rows 150-199 among others, whose `b` is damaged.
+ val expected = rows.filter { case (a, b, _) => Option(a).getOrElse(b) >= "a_15" }.map(_._3)
+ assert(expected.contains(150), "the filter must keep a row whose `b` is damaged")
+ assert(result == expected, s"got ${result.size} rows, expected ${expected.size}")
+ assert(givenUp == 1,
+ "the copy must have failed and given splicing up once, or this proves nothing; " +
+ s"logged $givenUp")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry reads the damaged row group whole over the pushed data filter's ranges, and only
+ // that one, since parquet keeps a column index store per block. What it must not give up is
+ // the pushed data filter's own page pruning, which a plain read of the same query keeps, nor
+ // the other row groups' narrowing. Dropping either would read more than it needs to.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warnings) = withLogged(partialReadWarning)(read(evenKeys))
+ assert(warnings == 1,
+ "the read must have discovered the damaged index, or this test proves nothing; got " +
+ warnings)
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ // The row groups around the damaged one keep the storage filter's narrowing too, so they
+ // return even keys alone.
+ val undamaged = filtered.filter(k => k <= 100L || k > 200L)
+ assert(undamaged.nonEmpty && undamaged.forall(_ % 2 == 0),
+ s"the other row groups must still be narrowed; got ${undamaged.filter(_ % 2 != 0)}")
+
+ // The WARN is reported once per split, however many row groups it costs.
+ val twiceDamaged = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(twiceDamaged, blockIdx = 0, columnIdx = 0)
+ breakOffsetIndex(twiceDamaged, blockIdx = 1, columnIdx = 0)
+ val (_, twiceWarned) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, evenKeys), twiceDamaged, ordinal = 0)
+ }
+ assert(twiceWarned == 1, s"two damaged row groups must be reported once; got $twiceWarned")
+ }
+ }
+
+ test("a kill that lands before an unreadable offset index is reported as a kill") {
+ // Parquet turns any IOException reading an offset index into `MissingOffsetIndexException`,
+ // a kill's interrupt included, so the catch asks for a kill before it takes a missing index.
+ // Here the kill lands while the last row group's one chunk is evaluated, after the chunk's own
+ // check, and with no row group after it. So only that catch can see it.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 2, columnIdx = 0)
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val killsAt202 = EqualTo(Remainder(
+ OnKey(BoundReference(0, LongType, nullable = false), at = 202L, killsTask = true),
+ Literal(2L)), Literal(0L))
+ val (thrown, warnings) = withLogged(partialReadWarning) {
+ asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ readLongs(formatReader(schema, pushed, Seq(killsAt202)), path, ordinal = 0)
+ }
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException], s"got $thrown")
+ assert(warnings == 0,
+ s"the kill must not be logged as an unreadable offset index; got $warnings")
+ }
+ }
+
+ test("a v2 file's DELTA_BYTE_ARRAY decimal column is read over scattered survivors") {
+ // Phase 2 skips rows inside a page whenever survivors scatter, where a plain read skips only
+ // when a pushed filter cuts a page. The parquet v2 writer stores a FIXED_LEN_BYTE_ARRAY
+ // decimal without a dictionary as DELTA_BYTE_ARRAY, whose reader could not skip until
+ // SPARK-59828, so this read failed where a plain read of the same file did not.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 1000).selectExpr("id AS k", "CAST(id * 1.25 AS DECIMAL(30, 2)) AS d"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.WRITER_VERSION -> "PARQUET_2_0"))
+ assert(encodingsOf(path, "d").contains(Encoding.DELTA_BYTE_ARRAY),
+ s"the fixture's whole point; got ${encodingsOf(path, "d")}")
+ val everyThirdRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(3L)), Literal(0L))
+ val schema = kvSchema(valueType = DecimalType(30, 2))
+ val (result, reader) = readAllWith(path, Seq("k", "d"),
+ createFilter(Seq(everyThirdRow), schema),
+ (b, i) => (b.column(0).getLong(i), b.column(1).getDecimal(i, 30, 2).toJavaBigDecimal))
+ try {
+ val expected = (0L until 1000L by 3L)
+ .map(k => (k, new java.math.BigDecimal(k).multiply(new java.math.BigDecimal("1.25"))
+ .setScale(2)))
+ assert(result == expected, s"got ${result.take(5)} (${result.size} rows)")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the survivor queue and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ val fileSchema = kvSchema()
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAll(path, filter, capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // Each survivor costs 9 bytes of long key and 40 of its own range. The sum passes the cap of
+ // 1024 at survivor 21, which releases the buffer, and the ranges alone at survivor 26, which
+ // gives the filter up. The capacity decides what the buffer holds when it is released. At
+ // 4096 that is one partly filled set of accumulators, and at 16 a full set in the queue as
+ // well. Both have to end with the queue empty and the filter given up. Keeping one without
+ // the other would leave phase 2 reading non-key columns while emit spliced keys from a queue
+ // holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAll(path, filter)
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSeq == Seq("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does, since the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers conjuncts under the relation's own column names") {
+ // The analyzer spells a reference the way the query did, so under case-insensitive resolution
+ // a bloom can name `_TMP_METADATA_ROW_INDEX` while the relation, and the schema the reader
+ // requests, call the column `_tmp_metadata_row_index`. The planner normalizes what it offers to
+ // the relation's names, so the format's gate sees the name the reader will see and declines.
+ // Without that the gate passed it, `ParquetStorageFilter.create` rejected it on the driver, and
+ // a query that runs with the conf off failed with it on.
+ withTempDir { dir =>
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", s"id AS $rowIndexName", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", rowIndexName, "v")
+ .queryExecution.optimizedPlan
+ def column(name: String): Attribute =
+ relation.output.find(_.name == name).getOrElse(fail(s"no $name in the relation output"))
+
+ def offered(key: Attribute): Seq[Expression] = {
+ val logical = LogicalFilter(
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ scanOf(physical).storageFilters
+ }
+
+ // As a control, an ordinary key spelled in another case is offered, under its own name.
+ val upperK = offered(column("k").withName("K"))
+ assert(upperK.size == 1, s"a bloom over `K` must reach the scan; got $upperK")
+ assert(upperK.head.references.map(_.name).toSet == Set("k"),
+ s"and it must carry the relation's name for the column; got ${upperK.head}")
+
+ val upperRowIndex =
+ offered(column(rowIndexName).withName(rowIndexName.toUpperCase(Locale.ROOT)))
+ assert(upperRowIndex.isEmpty,
+ s"a bloom over the row-index column must not be offered in any case; got $upperRowIndex")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers a conjunct that appears twice only once") {
+ // Spelled in two cases, a bloom over `k` normalizes to one conjunct, which the scan then takes
+ // once, as `afterScanFilters` does.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+ def bloomOn(key: Attribute): Expression =
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key)))
+ val logical = LogicalFilter(And(bloomOn(k), bloomOn(k.withName("K"))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ val offered = scanOf(physical).storageFilters
+ assert(offered.size == 1, s"the repeated conjunct must be offered once; got $offered")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for. A dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`, since `/` is floating-point division in Spark SQL, which would hand every
+ // key expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize, dictionary = dictionary)
+ }
+
+ // Every column chunk of the file, from its footer. The facts the tests below assert about a
+ // fixture are read off these rather than assumed from the write options.
+ private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] =
+ footerOf(filePath).getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq
+
+ private def footerOf(filePath: String): ParquetMetadata = {
+ val conf = spark.sessionState.newHadoopConf()
+ ParquetFooterReader.readFooter(
+ HadoopInputFile.fromPath(new Path(filePath), conf), ParquetMetadataConverter.NO_FILTER)
+ }
+
+ // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String, column: String = "k"): Set[Encoding] =
+ footerChunks(filePath).filter(_.getPath.toDotString == column)
+ .flatMap(_.getEncodings.asScala).toSet
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // builds the plain vectorized reader rather than the late-materialization one. Every read helper
+ // in this suite goes through here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need. The caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction. `onKeyReader`, if given, sees every key column reader phase 1 opens, see
+ // `KeyReaderHook`.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ if (storageFilter == null) {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } else if (onKeyReader != null) {
+ new KeyReaderHook(useOffHeap, capacity, storageFilter, onKeyReader)
+ } else {
+ new LateMaterializationParquetRecordReader(useOffHeap, capacity, storageFilter)
+ }
+ } { reader =>
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // A key value as a string in its *internal* representation, so the splicing path and the plain
+ // path can be compared without knowing the type and without external type conversion.
+ // `InternalRow.getAccessor` does the per-type read, which is why this takes the batch's row
+ // rather than the column vector.
+ private def renderValue(row: InternalRow, ordinal: Int, dt: DataType): String =
+ InternalRow.getAccessor(dt)(row, ordinal) match {
+ case null => "null"
+ // Two equal arrays are two objects, so the default toString would compare identities.
+ case bytes: Array[Byte] => bytes.mkString(",")
+ case value => value.toString
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path. Whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.getRow(i), 0, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage, one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why. There is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ // The byte-array copy and a set's growth go through `putByteArray`, `getBinary` and
+ // `reserve`, which off-heap vectors implement apart. So the variable-length types, strings,
+ // binaries and byte-array decimals, run off heap too, over a capacity of 4, so every set's
Review Comment:
Nit: not every set grows here. At capacity 4 a byte child starts at 16 bytes, so the string and binary arms' 5-byte keys grow a set only at its fourth value, and their last set, 3 keys on the plain arm and 1 on the dictionary arm, never does; only full sets and the 13-byte decimal arm's sets grow. Could this say a full set's byte child has to grow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4204521321
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1032 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A fixed-width key needs no
+ * factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. It is part of `keyFixedBytesPerRow`, counted
+ * twice, for its data byte and its null byte.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // decoders of their last page.
+ closeDataPages();
+ releaseRowGroupReaders();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (reportedMessages.add(OFFSET_INDEX_UNREADABLE)) {
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn(OFFSET_INDEX_UNREADABLE, MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_DECODING, true);
+ // The failed decode may have grown the scratch vectors, and phase 2 reads the same pages
+ // into vectors of its own. So capacity-sized ones replace them, and beside a plain read's
+ // own vectors the reader then holds one capacity-sized batch of key vectors.
+ closeAll(keyScratchVectors);
+ allocateKeyScratch();
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = predicate.eval(keyScratchBatch.getRow(r));
+ } catch (Throwable t) {
+ // Fall back on anything but a task kill or a fatal error. The predicate runs here without
+ // the conjuncts before it in the plan, so it can throw on a row a plain scan never
+ // evaluates it on (see `ParquetStorageFilter.isSupportedStorageFilter`). Giving the
+ // filter up returns every row the pushed filter kept, and the post-scan Filter decides
+ // them in its own order, raising the same error for a row it does evaluate.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK exception
+ // (`timestamp_seconds` of a decimal) or an `AssertionError`, and a Hive UDF throws a
+ // checked SparkException. Rethrowing would fail a query a plain read answers, or drop
+ // the rest of the file silently under `ignoreCorruptFiles`.
+ //
+ // A fatal error goes out even on a row an earlier conjunct of the plan would have
+ // rejected, so the reader never runs on after one the executor would take as fatal.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_EVALUATION, false);
+ return null;
+ }
+ if (!survives) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ if (currentKeyAccumulators == null) allocateKeyAccumulators();
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // The buffer is this reader's own and optional, so failing to fill it gives splicing up
+ // rather than the filter, and the memory goes back at once. What fails here:
+ // - a survivor vector that cannot grow;
+ // - a key value the predicate did not read on this row, such as `b` in
+ // `coalesce(a, b)`, since the copy decodes every key;
+ // - a bug of ours, which only the WARN then shows.
+ // Phase 2 reads the key columns the way a plain read does, so a value that cannot be
+ // decoded fails the read only where a plain read fails.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together.
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, charged per row as survivors
+ * arrive. A later set grows the way the first one does rather than being sized from the set
+ * before it. Sizing would save a long key's byte child its regrowth copies. But the memory it
+ * takes up front is a prediction, and charging it made what a row group is charged depend on the
+ * cap. A larger cap could then give splicing up where a smaller one kept it.
+ */
+ private void allocateKeyAccumulators() {
+ // Assigned before the loop, so an allocation failure part way through leaves the vectors
+ // allocated so far reachable for `abandonSplicing`.
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ currentKeyAccumulators[i] = newKeyVector(i);
+ }
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, which the caller has
+ * allocated, and hands them to the queue once they are full. Returns the bytes the row added,
+ * which the caller weighs against its budget.
+ */
+ private long appendSurvivorRowToAccumulators(int srcRow) {
+ long charged = 0L;
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final KeyColumn[] keys = keyColumns;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ KeyColumn key = keys[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ key.copier().copy(dst, dstRow, src, srcRow);
+ // Measured on the destination, because a dictionary-encoded source has no length of its
+ // own, its values being read through the dictionary.
+ if (key.variableLength()) {
+ charged += VARIABLE_LENGTH_BYTES_FACTOR * dst.getArrayLength(dstRow);
Review Comment:
Thanks, confirmed. Measuring the source, or `=` instead of `+=`, each kept all 122 tests green. The 32142 test now also runs on a dictionary-encoded file, which fails the first, and the 166 test on two string keys, which fails the second. Here, since both are small.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/StorageFilterPushdownBenchmark.scala:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import java.io.File
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetOutputFormat}
+import org.apache.parquet.hadoop.util.HadoopInputFile
+
+import org.apache.spark.SparkConf
+import org.apache.spark.benchmark.Benchmark
+import org.apache.spark.sql.{DataFrame, SparkSession}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
+import org.apache.spark.sql.catalyst.expressions.{BloomFilterMightContain, Literal, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.classic.ExpressionUtils.column
+import org.apache.spark.sql.execution.{FileSourceScanExec, SparkPlan}
+import org.apache.spark.sql.execution.datasources.parquet.StorageFilterMetrics
+import org.apache.spark.sql.functions.{bit_xor, col, count, lit, xxhash64}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.BinaryType
+import org.apache.spark.util.Utils
+
+/**
+ * Benchmark to measure storage-filter pushdown in the vectorized Parquet reader. Every case runs
+ * with `spark.sql.parquet.storageFilterPushdown.enabled` off and on, in two forms:
+ * - the join whose runtime bloom filter is the storage filter, which is the path the feature
+ * takes. The join's shuffle and sort are the same in both arms, and they grow with the rows
+ * that survive, so they dilute what the reader saves or costs;
+ * - a scan of the fact table alone under the same bloom filter, built once outside the timed
+ * body, which shows the reader's own share.
+ *
+ * Every case reads a fact table of about 100 MB in 34 row groups. The cases differ in four things:
+ * - the width of a row. One table has 50,000 rows of a long key and a 2 kB value that does not
+ * compress. The other has 1.4M rows of a long key and eight hashed long values;
+ * - the page size. Each table is written twice, in 8 kB pages and in parquet's default ones,
+ * 1 MB or 20,000 rows, whichever comes first. One surviving row makes the reader read its
+ * whole page, so this decides how much a survivor costs. A wide row fills a small page in a
+ * few values, while narrow ones still put about a thousand in it;
+ * - where the surviving keys fall. Keys from a contiguous range put the survivors in a few
+ * pages, so the reader skips the other row groups whole and the other pages of the row groups
+ * that hold them. Keys selected by their hash put survivors anywhere, so the reader skips
+ * only the pages none of them falls in, and decodes the rest range by range;
+ * - the share of the keys selected, 0.1%, 0.5%, 1% and 5%.
+ * Each section's title gives the row groups and rows per value page its table was written with.
+ * After each case it prints what the storage filter excluded, which does not depend on the
+ * machine.
+ *
+ * To run this benchmark:
+ * {{{
+ * 1. without sbt: bin/spark-submit --class <this class>
+ * --jars <spark core test jar>,<spark catalyst test jar> <spark sql test jar>
+ * 2. build/sbt "sql/Test/runMain <this class>"
+ * 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/Test/runMain <this class>"
+ * Results will be written to "benchmarks/StorageFilterPushdownBenchmark-results.txt".
+ * }}}
+ */
+object StorageFilterPushdownBenchmark extends SqlBasedBenchmark {
+
+ override def getSparkSession: SparkSession = {
+ val conf = new SparkConf()
+ .setAppName(this.getClass.getSimpleName)
+ // Since `spark.master` always exists, overrides this value
+ .set("spark.master", "local[1]")
+ .setIfMissing("spark.driver.memory", "3g")
+ .setIfMissing("spark.executor.memory", "3g")
+ .set(SQLConf.SHUFFLE_PARTITIONS.key, "1")
+ .set("spark.ui.enabled", "false")
+
+ SparkSession.builder().config(conf).getOrCreate()
+ }
+
+ // A fact table: its name and what its section title calls it, how many rows it has, how many of
+ // them a row group holds, and its value columns.
+ private case class Table(
+ name: String,
+ description: String,
+ numRows: Long,
+ rowsPerRowGroup: Int,
+ values: Seq[String])
+
+ // The values are hashed, so they neither compress nor dictionary-encode away, and the bytes a
+ // skipped page saves are real.
+ private val tables = Seq(
+ // 64 hashes of 32 bytes each.
+ Table("wide", "2 kB values", 50000L, 1500, Seq("unhex(concat_ws('', transform(" +
+ "sequence(0, 63), i -> sha2(concat(CAST(id AS STRING), '-', CAST(i AS STRING)), 256)))) " +
+ "AS v0")),
+ Table("narrow", "eight long values", 1400000L, 42000,
+ (0 until 8).map(i => s"xxhash64(id, $i) AS v$i")))
+
+ // The two ways each fact table is written: its layout's name, and the writer options that set
+ // its pages. Parquet checks a page's size only every 100 rows by default, so small pages need
+ // it checked after every row.
+ private val layouts = Seq(
+ "small_pages" -> Map(
+ ParquetOutputFormat.PAGE_SIZE -> "8192",
+ ParquetOutputFormat.MIN_ROW_COUNT_FOR_PAGE_SIZE_CHECK -> "1"),
+ "default_pages" -> Map.empty[String, String])
+
+ // Each share of the keys selected, in thousandths.
+ private val selected = Seq("0.1%" -> 1, "0.5%" -> 5, "1%" -> 10, "5%" -> 50)
+
+ private def prepareTables(dir: File): Unit = tables.foreach { table =>
+ val rows = spark.range(0, table.numRows, 1, 1).selectExpr("id AS k" +: table.values: _*)
+ layouts.foreach { case (layout, options) =>
+ val path = new File(dir, fact(table, layout)).getCanonicalPath
+ rows.write
+ .option(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT, table.rowsPerRowGroup.toString)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .options(options)
+ .parquet(path)
+ spark.read.parquet(path).createOrReplaceTempView(fact(table, layout))
+ }
+
+ def writeDimension(name: String, predicate: String): Unit = {
+ val path = new File(dir, name).getCanonicalPath
+ // `sel_id` exists for the query's filter on the dimension side, which is what makes the
+ // optimizer build a bloom filter from it at all.
+ spark.range(0, table.numRows, 1, 1).where(predicate).selectExpr("id AS k", "id AS sel_id")
+ .write.parquet(path)
+ spark.read.parquet(path).createOrReplaceTempView(name)
+ }
+ selected.foreach { case (label, perMille) =>
+ val start = table.numRows / 2
+ writeDimension(dimension(table, "clustered", label),
+ s"id >= $start AND id < ${start + table.numRows * perMille / 1000}")
+ writeDimension(dimension(table, "scattered", label),
+ s"pmod(xxhash64(id), 1000) < $perMille")
+ }
+ }
+
+ private def fact(table: Table, layout: String): String = s"fact_${table.name}_$layout"
+
+ private def dimension(table: Table, placement: String, label: String): String =
+ s"${table.name}_${placement}_${label.replace("%", "").replace(".", "_")}"
+
+ // How many row groups a fact table has, and how many rows its first value column's pages hold
+ // on average.
+ private def layoutOf(table: Table, layout: String): (Int, Long) = {
+ val file = new Path(spark.table(fact(table, layout)).inputFiles.head)
+ Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(file, spark.sessionState.newHadoopConf()))) { reader =>
+ val rowGroups = reader.getRowGroups.asScala
+ val valuePages = rowGroups.map { rowGroup =>
+ val value = rowGroup.getColumns.asScala.find(_.getPath.toDotString == "v0").get
+ reader.readOffsetIndex(value).getPageCount
+ }.sum
+ (rowGroups.size, table.numRows / valuePages)
+ }
+ }
+
+ private def query(fact: String, dimension: String): String =
+ s"SELECT f.* FROM $fact f JOIN $dimension d ON f.k = d.k WHERE d.sel_id >= 0"
+
+ // The fact table under the bloom filter the join builds from `dimension`, with no join. The
+ // bloom is built once, outside any timed body, the way `InjectRuntimeFilter` builds it for a
+ // creation side with no row count, which these temp views are.
+ private def scanOnly(fact: String, dimension: String): () => DataFrame = {
+ val keyHash = new XxHash64(Seq(UnresolvedAttribute("k")))
+ val bloom = spark.table(dimension)
+ .select(column(new BloomFilterAggregate(keyHash).toAggregateExpression()))
+ .head().getAs[Array[Byte]](0)
+ val mightContain = BloomFilterMightContain(Literal(bloom, BinaryType), keyHash)
+ () => spark.table(fact).where(column(mightContain))
+ }
+
+ private def factScan(plan: SparkPlan): FileSourceScanExec =
+ plan.collectFirst {
+ case scan: FileSourceScanExec if scan.output.exists(_.name == "v0") => scan
+ }.getOrElse(throw new IllegalStateException(s"No fact table scan in $plan"))
+
+ private def withPushdown[T](enabled: Boolean)(f: => T): T =
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> enabled.toString)(f)
+
+ // Asserts that `df` plans its bloom filter, and hands it to the fact scan exactly when
+ // pushdown is on, so neither arm can silently measure the other. Returns that scan.
+ private def checkedFactScan(df: DataFrame, enabled: Boolean): FileSourceScanExec = {
+ val plan = df.queryExecution.executedPlan
+ val hasBloom = plan.exists { node =>
+ node.expressions.exists(_.exists(_.isInstanceOf[BloomFilterMightContain]))
+ }
+ assert(hasBloom, s"The plan must have a bloom filter:\n${df.queryExecution}")
+ val scan = factScan(plan)
+ assert(scan.storageFilters.nonEmpty == enabled,
+ s"The fact scan must have a storage filter exactly when pushdown is on:\n$scan")
+ scan
+ }
+
+ // Runs `df` with pushdown off and on. Then, untimed, asserts that both arms return the same
+ // rows, counted and fingerprinted over every column, so a faster arm cannot be a wrong one.
+ // Returns the benchmark that timed it.
+ private def runOffAndOn(title: String, numRows: Long, df: () => DataFrame): Benchmark = {
+ val benchmark = new Benchmark(title, numRows, minNumIters = 3, output = output)
+ Seq(false, true).foreach { enabled =>
+ withPushdown(enabled)(checkedFactScan(df(), enabled))
+ benchmark.addCase(s"Storage filter pushdown ${if (enabled) "on" else "off"}") { _ =>
+ withPushdown(enabled)(df().noop())
+ }
+ }
+ benchmark.run()
+ def fingerprint(enabled: Boolean) = withPushdown(enabled) {
+ val rows = df()
+ val fingerprinted =
+ rows.select(count(lit(1)), bit_xor(xxhash64(rows.columns.map(col).toIndexedSeq: _*)))
+ checkedFactScan(fingerprinted, enabled)
+ fingerprinted.head()
+ }
+ val (off, on) = (fingerprint(enabled = false), fingerprint(enabled = true))
+ assert(off == on, s"$title: both arms must return the same rows; got $off and $on")
Review Comment:
Thanks, right. It ran once in [this workflow run](https://github.com/peter-toth/spark/actions/runs/37586579953), JDK 17 on `a442bf91ad4`, without committing results, and the check passed. It uses `collect().head` now.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1032 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A fixed-width key needs no
+ * factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. It is part of `keyFixedBytesPerRow`, counted
+ * twice, for its data byte and its null byte.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // decoders of their last page.
+ closeDataPages();
+ releaseRowGroupReaders();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (reportedMessages.add(OFFSET_INDEX_UNREADABLE)) {
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn(OFFSET_INDEX_UNREADABLE, MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_DECODING, true);
+ // The failed decode may have grown the scratch vectors, and phase 2 reads the same pages
+ // into vectors of its own. So capacity-sized ones replace them, and beside a plain read's
+ // own vectors the reader then holds one capacity-sized batch of key vectors.
+ closeAll(keyScratchVectors);
+ allocateKeyScratch();
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = predicate.eval(keyScratchBatch.getRow(r));
+ } catch (Throwable t) {
+ // Fall back on anything but a task kill or a fatal error. The predicate runs here without
+ // the conjuncts before it in the plan, so it can throw on a row a plain scan never
+ // evaluates it on (see `ParquetStorageFilter.isSupportedStorageFilter`). Giving the
+ // filter up returns every row the pushed filter kept, and the post-scan Filter decides
+ // them in its own order, raising the same error for a row it does evaluate.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK exception
+ // (`timestamp_seconds` of a decimal) or an `AssertionError`, and a Hive UDF throws a
+ // checked SparkException. Rethrowing would fail a query a plain read answers, or drop
+ // the rest of the file silently under `ignoreCorruptFiles`.
+ //
+ // A fatal error goes out even on a row an earlier conjunct of the plan would have
+ // rejected, so the reader never runs on after one the executor would take as fatal.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_EVALUATION, false);
+ return null;
+ }
+ if (!survives) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ if (currentKeyAccumulators == null) allocateKeyAccumulators();
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // The buffer is this reader's own and optional, so failing to fill it gives splicing up
+ // rather than the filter, and the memory goes back at once. What fails here:
+ // - a survivor vector that cannot grow;
+ // - a key value the predicate did not read on this row, such as `b` in
+ // `coalesce(a, b)`, since the copy decodes every key;
+ // - a bug of ours, which only the WARN then shows.
+ // Phase 2 reads the key columns the way a plain read does, so a value that cannot be
+ // decoded fails the read only where a plain read fails.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together.
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, charged per row as survivors
+ * arrive. A later set grows the way the first one does rather than being sized from the set
+ * before it. Sizing would save a long key's byte child its regrowth copies. But the memory it
Review Comment:
Done, in your words, split into sentences. The second sentence of the `DEFAULT_CHILD_BYTES_PER_ROW` javadoc is gone.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3708 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // The native address of an off-heap vector's values, 0 once it is freed. Only for a fixed-width
+ // vector. A string or binary one keeps its bytes in a child and reports 0 even while live.
+ private def address(vector: Any): Long =
+ vector.asInstanceOf[OffHeapColumnVector].valuesNativeAddress()
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t @ (_: VirtualMachineError | _: LinkageError) =>
+ fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`, and returns the rows.
+ private def readLaterSet(path: String, cap: Long): Seq[(String, String)] = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString),
+ capacity = 4)
+ reader.close()
+ rows
+ }
+
+ test("a later accumulator set of a variable-length key is charged per value") {
+ // Six survivors over a capacity of 4 fill one set of four 1000-byte keys and a second of two
+ // 2000-byte ones. Every set grows as its values arrive, the way the first one does. Each value
+ // is charged four times its bytes plus its row's fixed part of 17, and the one range adds 40.
+ // So the row group is charged 4 * (4000 + 17) + 2 * (8000 + 17) + 40 = 32142 bytes. A cap of
+ // 32142 keeps splicing and one of 32141 gives it up, which a missing or a double charge would
+ // each get wrong. And the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ assertSpliceCharge(32142L, rows = 6) { cap =>
+ readUnderCap(path, laterSetSchema, laterSetFilter, cap, batchSize = 4)
+ }
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val rows = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ }
+ }
+ }
+
+ test("what a row group of scattered survivors is charged does not depend on the cap") {
+ // Keys 9 to 12 fill the first set over a capacity of 4, and keys 14 and 17 come scattered, each
+ // in a range of its own. Every set grows as values arrive and is charged per value, so the row
+ // group's charge is one number: the first set's 16068, three ranges of 40 and the second
+ // set's 8034, 24222 in all. A cap of it splices and one byte under it gives splicing up.
+ // Larger caps splice too, which a few of them check, among them 32216 to 32221. Those gave
+ // splicing up when a later set was sized from the one before it with no room for its ranges.
+ // The second set is not sized at all, so its byte child grows to 2000 bytes for its two
+ // 1000-byte values, where sizing it from the first set would have reserved 8000.
+ withTempDir { dir =>
+ val (path, _) = writeLaterSetFile(dir, lastTwoBytes = 1000)
+ val prefix = Substring(BoundReference(0, StringType, nullable = true), Literal(1), Literal(4))
+ val keys = Seq("0009", "0010", "0011", "0012", "0014", "0017").map(Literal(_))
+ def readUnder(cap: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, Seq(In(prefix, keys)), cap, batchSize = 4)
+ assertSpliceCharge(24222L, rows = 6)(readUnder)
+ val (_, uncapped) = readUnder("64MB")
+ Seq(32215L, 32216L, 32221L, 32222L).foreach { cap =>
+ val (rows, bytes) = readUnder(s"${cap}b")
+ assert(rows == 6 && bytes == uncapped,
+ s"a cap of $cap must keep the six rows and splicing; got $rows rows")
+ }
+ val (children, reader) = readAllWith(path, Seq("k", "v"),
+ createFilter(Seq(In(prefix, keys)), laterSetSchema),
+ (b, _) => ParquetTestAccess.byteChildCapacity(
+ b.column(0).asInstanceOf[WritableColumnVector]),
+ capacity = 4)
+ reader.close()
+ assert(children.drop(4).distinct == Seq(2000),
+ s"the second set's byte child must grow as values arrive; got $children")
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ assertSpliceCharge(166L, rows = 7) { cap =>
+ readUnderCap(path, schema, storageFilters, cap, batchSize = 16)
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warnings) = withLogged(partialReadWarning) {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(warnings == 0,
+ s"the footer check must get there before the exception path; got $warnings")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warnings) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(warnings == 0, "the footer check must decline the row group rather than the read " +
+ s"discovering it; got $warnings")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize = 64L * 1024 * 1024,
+ options = Map(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT -> "100"))
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ // The rows a reader hands back before it throws, what it throws, and, for the filtering
+ // reader, whether its scratch vectors were replaced on the way and the old ones freed. Both
+ // read off heap, where an old scratch vector left open would leak native memory.
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable, Boolean) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(true, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(true, 16, storageFilter)
+ }
+ def scratch: Seq[WritableColumnVector] = reader match {
+ case filtering: LateMaterializationParquetRecordReader =>
+ ParquetTestAccess.keyScratchVectors(filtering)
+ case _ => Nil
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val scratchBefore = scratch
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ val replacedAndFreed = scratch.zip(scratchBefore).exists { case (now, before) =>
+ (now ne before) && address(before) == 0
+ }
+ (read.toSeq, error, replacedAndFreed)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError, _) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError, scratchReplaced) =
+ readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ // A failed decode can leave the scratch vectors grown. Phase 2 reads the same pages into
+ // vectors of its own, so the reader replaces them with capacity-sized ones rather than keep
+ // the grown ones for the rest of the split.
+ assert(scratchReplaced,
+ "the decoding fallback must replace the scratch vectors and free the old ones")
+
+ // A kill that shows only at the decoding catch, after the per-chunk check has passed, is
+ // seen there. Without that catch's own check the row group would fall back and phase 2
+ // would fail on the page instead.
+ val killedAtTheCatch = spy(TaskContext.empty())
+ doAnswer((_: InvocationOnMock) => Boolean.box(Thread.currentThread().getStackTrace
+ .exists(_.getMethodName.startsWith("rethrowIfMustPropagate"))))
+ .when(killedAtTheCatch).isInterrupted()
+ val (_, killed, _) = TaskContext.withTaskContext(killedAtTheCatch) {
+ readUntilFailure(keyAtLeastFilter(500L))
+ }
+ assert(killed.isInstanceOf[SparkException] && killed.getMessage.contains("task kill") &&
+ killed.getCause != null && killed.getCause.getClass == plainError.getClass,
+ s"the kill must go out from the decoding catch, around the decoding error; got $killed")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+
+ test("a key vector that cannot grow, or a codec's InternalError, gives the row group up") {
+ // Both are fatal-looking errors that decoding the key pages can raise, and the reader falls
+ // back on both. A scratch vector that cannot grow is phase 1's own, and phase 2 then reads
+ // what a plain read reads. An `InternalError` from decoding is how a codec like hadoop-lzo
+ // reports a corrupt page, which the plain read of the same pages meets again. Raised from the
+ // filter instead, the first goes out as its OutOfMemoryError and the second in an
+ // `UnknownError`. Off heap, the scratch vectors the failed decode used must be freed, and the
+ // second row group's phase 1 runs over the ones that replace them.
+ withTempDir { dir =>
+ val rows = (0L until 40L).map(i => (i, i.toString))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "20"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(20L, 20L),
+ "the fixture must be two row groups of 20 rows")
+ Seq(
+ new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve")),
+ new InternalError("codec")).foreach { error =>
+ var failedScratch: Seq[WritableColumnVector] = Nil
+ // Only the first row group's key reader fails.
+ val failing: KeyReaderHook.OnKeyReader = (reader, keyReader) => {
+ val scratch = ParquetTestAccess.keyScratchVectors(
+ reader.asInstanceOf[LateMaterializationParquetRecordReader])
+ if (failedScratch.nonEmpty) {
+ // Checked before phase 1 writes into them, since a freed one would crash the JVM.
+ assert(scratch.forall(v => !failedScratch.exists(_ eq v) && address(v) != 0),
+ "the second row group must decode into new scratch vectors")
+ keyReader
+ } else {
+ failedScratch = scratch
+ val failingReader = spy(keyReader)
+ doThrow(error).when(failingReader).readBatch(anyInt(), any(), any(), any())
+ failingReader
+ }
+ }
+ val ((result, reader), warnings) = try {
+ withLogged(decodingGivenUp) {
+ readAll(path, keyAtLeastFilter(30L), useOffHeap = true, onKeyReader = failing)
+ }
+ } catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case e: VirtualMachineError => fail(s"the reader must fall back on $error; got $e", e)
+ }
+ try {
+ assert(result == rows.take(20) ++ rows.drop(30),
+ s"the first row group must come back whole and the second filtered on $error; got " +
+ s"${result.map(_._1)}")
+ assert(warnings == 1, s"the decoding fallback must report $error; got $warnings")
+ assert(failedScratch.forall(v => address(v) == 0),
+ s"the scratch vectors of the failed decode must be freed on $error")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("a key value the filter does not read and cannot be decoded gives splicing up") {
+ // The filter need not read every key on every row, as `coalesce(a, b)` reads `b` only where
+ // `a` is null. The copy into the survivor buffer reads every key, so it can meet a value no
+ // plain read decodes. A dictionary-encoded column decodes a value only when it is read, so a
+ // dictionary that holds fewer values than its ids point at is such a value. Copying it gives
+ // splicing up for the row group, and phase 2 reads the key columns the way a plain read does.
+ // So the read answers where a plain one does, and the filter still applies.
+ withTempDir { dir =>
+ // Two copies of each `b`, so parquet keeps its dictionary. Cut to 64 values, it leaves `b`
+ // undecodable from row 128 on. `a` is null only before that, so the filter never reads a
+ // damaged `b`.
+ val rows = (0 until 200).map { i =>
+ (if (i < 100 && i % 5 == 0) null else s"a_$i", s"b_${i / 2}", i)
+ }
+ val path = writeSingleParquetFile(dir, rows.toDF("a", "b", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE, dictionary = true)
+ assert(encodingsOf(path, "b").exists(_.usesDictionary),
+ s"the fixture's whole point; got ${encodingsOf(path, "b")}")
+ shrinkDictionary(path, column = "b", entries = 64)
+ val columns = Seq("a", "b", "v")
+ intercept[ArrayIndexOutOfBoundsException] {
+ readAllWith(path, columns, null, (b, i) => b.column(1).getUTF8String(i).toString)._2
+ .close()
+ }
+
+ val schema = StructType(Seq(
+ StructField("a", StringType), StructField("b", StringType), StructField("v", IntegerType)))
+ val coalesced = Coalesce(Seq(BoundReference(0, StringType, nullable = true),
+ BoundReference(1, StringType, nullable = true)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(coalesced, Literal.create("a_15", StringType))), schema)
+ val ((result, reader), givenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, columns, filter, (b, i) => b.column(2).getInt(i), capacity = 16)
+ }
+ try {
+ // Keeps rows 150-199 among others, whose `b` is damaged.
+ val expected = rows.filter { case (a, b, _) => Option(a).getOrElse(b) >= "a_15" }.map(_._3)
+ assert(expected.contains(150), "the filter must keep a row whose `b` is damaged")
+ assert(result == expected, s"got ${result.size} rows, expected ${expected.size}")
+ assert(givenUp == 1,
+ "the copy must have failed and given splicing up once, or this proves nothing; " +
+ s"logged $givenUp")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry reads the damaged row group whole over the pushed data filter's ranges, and only
+ // that one, since parquet keeps a column index store per block. What it must not give up is
+ // the pushed data filter's own page pruning, which a plain read of the same query keeps, nor
+ // the other row groups' narrowing. Dropping either would read more than it needs to.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warnings) = withLogged(partialReadWarning)(read(evenKeys))
+ assert(warnings == 1,
+ "the read must have discovered the damaged index, or this test proves nothing; got " +
+ warnings)
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ // The row groups around the damaged one keep the storage filter's narrowing too, so they
+ // return even keys alone.
+ val undamaged = filtered.filter(k => k <= 100L || k > 200L)
+ assert(undamaged.nonEmpty && undamaged.forall(_ % 2 == 0),
+ s"the other row groups must still be narrowed; got ${undamaged.filter(_ % 2 != 0)}")
+
+ // The WARN is reported once per split, however many row groups it costs.
+ val twiceDamaged = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(twiceDamaged, blockIdx = 0, columnIdx = 0)
+ breakOffsetIndex(twiceDamaged, blockIdx = 1, columnIdx = 0)
+ val (_, twiceWarned) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, evenKeys), twiceDamaged, ordinal = 0)
+ }
+ assert(twiceWarned == 1, s"two damaged row groups must be reported once; got $twiceWarned")
+ }
+ }
+
+ test("a kill that lands before an unreadable offset index is reported as a kill") {
+ // Parquet turns any IOException reading an offset index into `MissingOffsetIndexException`,
+ // a kill's interrupt included, so the catch asks for a kill before it takes a missing index.
+ // Here the kill lands while the last row group's one chunk is evaluated, after the chunk's own
+ // check, and with no row group after it. So only that catch can see it.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 2, columnIdx = 0)
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val killsAt202 = EqualTo(Remainder(
+ OnKey(BoundReference(0, LongType, nullable = false), at = 202L, killsTask = true),
+ Literal(2L)), Literal(0L))
+ val (thrown, warnings) = withLogged(partialReadWarning) {
+ asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ readLongs(formatReader(schema, pushed, Seq(killsAt202)), path, ordinal = 0)
+ }
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException], s"got $thrown")
+ assert(warnings == 0,
+ s"the kill must not be logged as an unreadable offset index; got $warnings")
+ }
+ }
+
+ test("a v2 file's DELTA_BYTE_ARRAY decimal column is read over scattered survivors") {
+ // Phase 2 skips rows inside a page whenever survivors scatter, where a plain read skips only
+ // when a pushed filter cuts a page. The parquet v2 writer stores a FIXED_LEN_BYTE_ARRAY
+ // decimal without a dictionary as DELTA_BYTE_ARRAY, whose reader could not skip until
+ // SPARK-59828, so this read failed where a plain read of the same file did not.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 1000).selectExpr("id AS k", "CAST(id * 1.25 AS DECIMAL(30, 2)) AS d"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.WRITER_VERSION -> "PARQUET_2_0"))
+ assert(encodingsOf(path, "d").contains(Encoding.DELTA_BYTE_ARRAY),
+ s"the fixture's whole point; got ${encodingsOf(path, "d")}")
+ val everyThirdRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(3L)), Literal(0L))
+ val schema = kvSchema(valueType = DecimalType(30, 2))
+ val (result, reader) = readAllWith(path, Seq("k", "d"),
+ createFilter(Seq(everyThirdRow), schema),
+ (b, i) => (b.column(0).getLong(i), b.column(1).getDecimal(i, 30, 2).toJavaBigDecimal))
+ try {
+ val expected = (0L until 1000L by 3L)
+ .map(k => (k, new java.math.BigDecimal(k).multiply(new java.math.BigDecimal("1.25"))
+ .setScale(2)))
+ assert(result == expected, s"got ${result.take(5)} (${result.size} rows)")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the survivor queue and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ val fileSchema = kvSchema()
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAll(path, filter, capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // Each survivor costs 9 bytes of long key and 40 of its own range. The sum passes the cap of
+ // 1024 at survivor 21, which releases the buffer, and the ranges alone at survivor 26, which
+ // gives the filter up. The capacity decides what the buffer holds when it is released. At
+ // 4096 that is one partly filled set of accumulators, and at 16 a full set in the queue as
+ // well. Both have to end with the queue empty and the filter given up. Keeping one without
+ // the other would leave phase 2 reading non-key columns while emit spliced keys from a queue
+ // holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAll(path, filter)
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSeq == Seq("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does, since the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers conjuncts under the relation's own column names") {
+ // The analyzer spells a reference the way the query did, so under case-insensitive resolution
+ // a bloom can name `_TMP_METADATA_ROW_INDEX` while the relation, and the schema the reader
+ // requests, call the column `_tmp_metadata_row_index`. The planner normalizes what it offers to
+ // the relation's names, so the format's gate sees the name the reader will see and declines.
+ // Without that the gate passed it, `ParquetStorageFilter.create` rejected it on the driver, and
+ // a query that runs with the conf off failed with it on.
+ withTempDir { dir =>
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", s"id AS $rowIndexName", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", rowIndexName, "v")
+ .queryExecution.optimizedPlan
+ def column(name: String): Attribute =
+ relation.output.find(_.name == name).getOrElse(fail(s"no $name in the relation output"))
+
+ def offered(key: Attribute): Seq[Expression] = {
+ val logical = LogicalFilter(
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ scanOf(physical).storageFilters
+ }
+
+ // As a control, an ordinary key spelled in another case is offered, under its own name.
+ val upperK = offered(column("k").withName("K"))
+ assert(upperK.size == 1, s"a bloom over `K` must reach the scan; got $upperK")
+ assert(upperK.head.references.map(_.name).toSet == Set("k"),
+ s"and it must carry the relation's name for the column; got ${upperK.head}")
+
+ val upperRowIndex =
+ offered(column(rowIndexName).withName(rowIndexName.toUpperCase(Locale.ROOT)))
+ assert(upperRowIndex.isEmpty,
+ s"a bloom over the row-index column must not be offered in any case; got $upperRowIndex")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers a conjunct that appears twice only once") {
+ // Spelled in two cases, a bloom over `k` normalizes to one conjunct, which the scan then takes
+ // once, as `afterScanFilters` does.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+ def bloomOn(key: Attribute): Expression =
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key)))
+ val logical = LogicalFilter(And(bloomOn(k), bloomOn(k.withName("K"))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ val offered = scanOf(physical).storageFilters
+ assert(offered.size == 1, s"the repeated conjunct must be offered once; got $offered")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for. A dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`, since `/` is floating-point division in Spark SQL, which would hand every
+ // key expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize, dictionary = dictionary)
+ }
+
+ // Every column chunk of the file, from its footer. The facts the tests below assert about a
+ // fixture are read off these rather than assumed from the write options.
+ private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] =
+ footerOf(filePath).getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq
+
+ private def footerOf(filePath: String): ParquetMetadata = {
+ val conf = spark.sessionState.newHadoopConf()
+ ParquetFooterReader.readFooter(
+ HadoopInputFile.fromPath(new Path(filePath), conf), ParquetMetadataConverter.NO_FILTER)
+ }
+
+ // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String, column: String = "k"): Set[Encoding] =
+ footerChunks(filePath).filter(_.getPath.toDotString == column)
+ .flatMap(_.getEncodings.asScala).toSet
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // builds the plain vectorized reader rather than the late-materialization one. Every read helper
+ // in this suite goes through here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need. The caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction. `onKeyReader`, if given, sees every key column reader phase 1 opens, see
+ // `KeyReaderHook`.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ if (storageFilter == null) {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } else if (onKeyReader != null) {
+ new KeyReaderHook(useOffHeap, capacity, storageFilter, onKeyReader)
+ } else {
+ new LateMaterializationParquetRecordReader(useOffHeap, capacity, storageFilter)
+ }
+ } { reader =>
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // A key value as a string in its *internal* representation, so the splicing path and the plain
+ // path can be compared without knowing the type and without external type conversion.
+ // `InternalRow.getAccessor` does the per-type read, which is why this takes the batch's row
+ // rather than the column vector.
+ private def renderValue(row: InternalRow, ordinal: Int, dt: DataType): String =
+ InternalRow.getAccessor(dt)(row, ordinal) match {
+ case null => "null"
+ // Two equal arrays are two objects, so the default toString would compare identities.
+ case bytes: Array[Byte] => bytes.mkString(",")
+ case value => value.toString
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path. Whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.getRow(i), 0, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage, one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why. There is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ // The byte-array copy and a set's growth go through `putByteArray`, `getBinary` and
+ // `reserve`, which off-heap vectors implement apart. So the variable-length types, strings,
+ // binaries and byte-array decimals, run off heap too, over a capacity of 4, so every set's
Review Comment:
Done, it says a full set's byte child has to grow.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3708 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // The native address of an off-heap vector's values, 0 once it is freed. Only for a fixed-width
+ // vector. A string or binary one keeps its bytes in a child and reports 0 even while live.
+ private def address(vector: Any): Long =
+ vector.asInstanceOf[OffHeapColumnVector].valuesNativeAddress()
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t @ (_: VirtualMachineError | _: LinkageError) =>
+ fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`, and returns the rows.
+ private def readLaterSet(path: String, cap: Long): Seq[(String, String)] = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString),
+ capacity = 4)
+ reader.close()
+ rows
+ }
+
+ test("a later accumulator set of a variable-length key is charged per value") {
+ // Six survivors over a capacity of 4 fill one set of four 1000-byte keys and a second of two
+ // 2000-byte ones. Every set grows as its values arrive, the way the first one does. Each value
+ // is charged four times its bytes plus its row's fixed part of 17, and the one range adds 40.
+ // So the row group is charged 4 * (4000 + 17) + 2 * (8000 + 17) + 40 = 32142 bytes. A cap of
+ // 32142 keeps splicing and one of 32141 gives it up, which a missing or a double charge would
+ // each get wrong. And the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ assertSpliceCharge(32142L, rows = 6) { cap =>
+ readUnderCap(path, laterSetSchema, laterSetFilter, cap, batchSize = 4)
+ }
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val rows = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ }
+ }
+ }
+
+ test("what a row group of scattered survivors is charged does not depend on the cap") {
+ // Keys 9 to 12 fill the first set over a capacity of 4, and keys 14 and 17 come scattered, each
+ // in a range of its own. Every set grows as values arrive and is charged per value, so the row
+ // group's charge is one number: the first set's 16068, three ranges of 40 and the second
+ // set's 8034, 24222 in all. A cap of it splices and one byte under it gives splicing up.
+ // Larger caps splice too, which a few of them check, among them 32216 to 32221. Those gave
+ // splicing up when a later set was sized from the one before it with no room for its ranges.
+ // The second set is not sized at all, so its byte child grows to 2000 bytes for its two
+ // 1000-byte values, where sizing it from the first set would have reserved 8000.
+ withTempDir { dir =>
+ val (path, _) = writeLaterSetFile(dir, lastTwoBytes = 1000)
+ val prefix = Substring(BoundReference(0, StringType, nullable = true), Literal(1), Literal(4))
+ val keys = Seq("0009", "0010", "0011", "0012", "0014", "0017").map(Literal(_))
+ def readUnder(cap: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, Seq(In(prefix, keys)), cap, batchSize = 4)
+ assertSpliceCharge(24222L, rows = 6)(readUnder)
+ val (_, uncapped) = readUnder("64MB")
+ Seq(32215L, 32216L, 32221L, 32222L).foreach { cap =>
+ val (rows, bytes) = readUnder(s"${cap}b")
+ assert(rows == 6 && bytes == uncapped,
+ s"a cap of $cap must keep the six rows and splicing; got $rows rows")
+ }
+ val (children, reader) = readAllWith(path, Seq("k", "v"),
+ createFilter(Seq(In(prefix, keys)), laterSetSchema),
+ (b, _) => ParquetTestAccess.byteChildCapacity(
+ b.column(0).asInstanceOf[WritableColumnVector]),
+ capacity = 4)
+ reader.close()
+ assert(children.drop(4).distinct == Seq(2000),
+ s"the second set's byte child must grow as values arrive; got $children")
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ assertSpliceCharge(166L, rows = 7) { cap =>
+ readUnderCap(path, schema, storageFilters, cap, batchSize = 16)
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warnings) = withLogged(partialReadWarning) {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(warnings == 0,
+ s"the footer check must get there before the exception path; got $warnings")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warnings) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(warnings == 0, "the footer check must decline the row group rather than the read " +
+ s"discovering it; got $warnings")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize = 64L * 1024 * 1024,
+ options = Map(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT -> "100"))
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ // The rows a reader hands back before it throws, what it throws, and, for the filtering
+ // reader, whether its scratch vectors were replaced on the way and the old ones freed. Both
+ // read off heap, where an old scratch vector left open would leak native memory.
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable, Boolean) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(true, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(true, 16, storageFilter)
+ }
+ def scratch: Seq[WritableColumnVector] = reader match {
+ case filtering: LateMaterializationParquetRecordReader =>
+ ParquetTestAccess.keyScratchVectors(filtering)
+ case _ => Nil
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val scratchBefore = scratch
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ val replacedAndFreed = scratch.zip(scratchBefore).exists { case (now, before) =>
+ (now ne before) && address(before) == 0
+ }
+ (read.toSeq, error, replacedAndFreed)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError, _) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError, scratchReplaced) =
+ readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ // A failed decode can leave the scratch vectors grown. Phase 2 reads the same pages into
+ // vectors of its own, so the reader replaces them with capacity-sized ones rather than keep
+ // the grown ones for the rest of the split.
+ assert(scratchReplaced,
+ "the decoding fallback must replace the scratch vectors and free the old ones")
+
+ // A kill that shows only at the decoding catch, after the per-chunk check has passed, is
+ // seen there. Without that catch's own check the row group would fall back and phase 2
+ // would fail on the page instead.
+ val killedAtTheCatch = spy(TaskContext.empty())
+ doAnswer((_: InvocationOnMock) => Boolean.box(Thread.currentThread().getStackTrace
+ .exists(_.getMethodName.startsWith("rethrowIfMustPropagate"))))
+ .when(killedAtTheCatch).isInterrupted()
+ val (_, killed, _) = TaskContext.withTaskContext(killedAtTheCatch) {
+ readUntilFailure(keyAtLeastFilter(500L))
+ }
+ assert(killed.isInstanceOf[SparkException] && killed.getMessage.contains("task kill") &&
+ killed.getCause != null && killed.getCause.getClass == plainError.getClass,
+ s"the kill must go out from the decoding catch, around the decoding error; got $killed")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+
+ test("a key vector that cannot grow, or a codec's InternalError, gives the row group up") {
+ // Both are fatal-looking errors that decoding the key pages can raise, and the reader falls
+ // back on both. A scratch vector that cannot grow is phase 1's own, and phase 2 then reads
+ // what a plain read reads. An `InternalError` from decoding is how a codec like hadoop-lzo
+ // reports a corrupt page, which the plain read of the same pages meets again. Raised from the
+ // filter instead, the first goes out as its OutOfMemoryError and the second in an
+ // `UnknownError`. Off heap, the scratch vectors the failed decode used must be freed, and the
+ // second row group's phase 1 runs over the ones that replace them.
+ withTempDir { dir =>
+ val rows = (0L until 40L).map(i => (i, i.toString))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "20"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(20L, 20L),
+ "the fixture must be two row groups of 20 rows")
+ Seq(
+ new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve")),
+ new InternalError("codec")).foreach { error =>
+ var failedScratch: Seq[WritableColumnVector] = Nil
+ // Only the first row group's key reader fails.
+ val failing: KeyReaderHook.OnKeyReader = (reader, keyReader) => {
+ val scratch = ParquetTestAccess.keyScratchVectors(
+ reader.asInstanceOf[LateMaterializationParquetRecordReader])
+ if (failedScratch.nonEmpty) {
+ // Checked before phase 1 writes into them, since a freed one would crash the JVM.
+ assert(scratch.forall(v => !failedScratch.exists(_ eq v) && address(v) != 0),
+ "the second row group must decode into new scratch vectors")
+ keyReader
+ } else {
+ failedScratch = scratch
+ val failingReader = spy(keyReader)
+ doThrow(error).when(failingReader).readBatch(anyInt(), any(), any(), any())
+ failingReader
+ }
+ }
+ val ((result, reader), warnings) = try {
+ withLogged(decodingGivenUp) {
+ readAll(path, keyAtLeastFilter(30L), useOffHeap = true, onKeyReader = failing)
+ }
+ } catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case e: VirtualMachineError => fail(s"the reader must fall back on $error; got $e", e)
+ }
+ try {
+ assert(result == rows.take(20) ++ rows.drop(30),
+ s"the first row group must come back whole and the second filtered on $error; got " +
+ s"${result.map(_._1)}")
+ assert(warnings == 1, s"the decoding fallback must report $error; got $warnings")
+ assert(failedScratch.forall(v => address(v) == 0),
+ s"the scratch vectors of the failed decode must be freed on $error")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("a key value the filter does not read and cannot be decoded gives splicing up") {
+ // The filter need not read every key on every row, as `coalesce(a, b)` reads `b` only where
+ // `a` is null. The copy into the survivor buffer reads every key, so it can meet a value no
+ // plain read decodes. A dictionary-encoded column decodes a value only when it is read, so a
+ // dictionary that holds fewer values than its ids point at is such a value. Copying it gives
+ // splicing up for the row group, and phase 2 reads the key columns the way a plain read does.
+ // So the read answers where a plain one does, and the filter still applies.
+ withTempDir { dir =>
+ // Two copies of each `b`, so parquet keeps its dictionary. Cut to 64 values, it leaves `b`
+ // undecodable from row 128 on. `a` is null only before that, so the filter never reads a
+ // damaged `b`.
+ val rows = (0 until 200).map { i =>
+ (if (i < 100 && i % 5 == 0) null else s"a_$i", s"b_${i / 2}", i)
+ }
+ val path = writeSingleParquetFile(dir, rows.toDF("a", "b", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE, dictionary = true)
+ assert(encodingsOf(path, "b").exists(_.usesDictionary),
+ s"the fixture's whole point; got ${encodingsOf(path, "b")}")
+ shrinkDictionary(path, column = "b", entries = 64)
+ val columns = Seq("a", "b", "v")
+ intercept[ArrayIndexOutOfBoundsException] {
+ readAllWith(path, columns, null, (b, i) => b.column(1).getUTF8String(i).toString)._2
+ .close()
+ }
+
+ val schema = StructType(Seq(
+ StructField("a", StringType), StructField("b", StringType), StructField("v", IntegerType)))
+ val coalesced = Coalesce(Seq(BoundReference(0, StringType, nullable = true),
+ BoundReference(1, StringType, nullable = true)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(coalesced, Literal.create("a_15", StringType))), schema)
+ val ((result, reader), givenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, columns, filter, (b, i) => b.column(2).getInt(i), capacity = 16)
+ }
+ try {
+ // Keeps rows 150-199 among others, whose `b` is damaged.
+ val expected = rows.filter { case (a, b, _) => Option(a).getOrElse(b) >= "a_15" }.map(_._3)
+ assert(expected.contains(150), "the filter must keep a row whose `b` is damaged")
+ assert(result == expected, s"got ${result.size} rows, expected ${expected.size}")
+ assert(givenUp == 1,
+ "the copy must have failed and given splicing up once, or this proves nothing; " +
+ s"logged $givenUp")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry reads the damaged row group whole over the pushed data filter's ranges, and only
+ // that one, since parquet keeps a column index store per block. What it must not give up is
+ // the pushed data filter's own page pruning, which a plain read of the same query keeps, nor
+ // the other row groups' narrowing. Dropping either would read more than it needs to.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warnings) = withLogged(partialReadWarning)(read(evenKeys))
+ assert(warnings == 1,
+ "the read must have discovered the damaged index, or this test proves nothing; got " +
+ warnings)
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ // The row groups around the damaged one keep the storage filter's narrowing too, so they
+ // return even keys alone.
+ val undamaged = filtered.filter(k => k <= 100L || k > 200L)
+ assert(undamaged.nonEmpty && undamaged.forall(_ % 2 == 0),
+ s"the other row groups must still be narrowed; got ${undamaged.filter(_ % 2 != 0)}")
+
+ // The WARN is reported once per split, however many row groups it costs.
+ val twiceDamaged = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(twiceDamaged, blockIdx = 0, columnIdx = 0)
+ breakOffsetIndex(twiceDamaged, blockIdx = 1, columnIdx = 0)
+ val (_, twiceWarned) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, evenKeys), twiceDamaged, ordinal = 0)
+ }
+ assert(twiceWarned == 1, s"two damaged row groups must be reported once; got $twiceWarned")
+ }
+ }
+
+ test("a kill that lands before an unreadable offset index is reported as a kill") {
+ // Parquet turns any IOException reading an offset index into `MissingOffsetIndexException`,
+ // a kill's interrupt included, so the catch asks for a kill before it takes a missing index.
+ // Here the kill lands while the last row group's one chunk is evaluated, after the chunk's own
+ // check, and with no row group after it. So only that catch can see it.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 2, columnIdx = 0)
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val killsAt202 = EqualTo(Remainder(
+ OnKey(BoundReference(0, LongType, nullable = false), at = 202L, killsTask = true),
+ Literal(2L)), Literal(0L))
+ val (thrown, warnings) = withLogged(partialReadWarning) {
+ asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ readLongs(formatReader(schema, pushed, Seq(killsAt202)), path, ordinal = 0)
+ }
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException], s"got $thrown")
+ assert(warnings == 0,
+ s"the kill must not be logged as an unreadable offset index; got $warnings")
+ }
+ }
+
+ test("a v2 file's DELTA_BYTE_ARRAY decimal column is read over scattered survivors") {
+ // Phase 2 skips rows inside a page whenever survivors scatter, where a plain read skips only
+ // when a pushed filter cuts a page. The parquet v2 writer stores a FIXED_LEN_BYTE_ARRAY
+ // decimal without a dictionary as DELTA_BYTE_ARRAY, whose reader could not skip until
+ // SPARK-59828, so this read failed where a plain read of the same file did not.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 1000).selectExpr("id AS k", "CAST(id * 1.25 AS DECIMAL(30, 2)) AS d"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.WRITER_VERSION -> "PARQUET_2_0"))
+ assert(encodingsOf(path, "d").contains(Encoding.DELTA_BYTE_ARRAY),
+ s"the fixture's whole point; got ${encodingsOf(path, "d")}")
+ val everyThirdRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(3L)), Literal(0L))
+ val schema = kvSchema(valueType = DecimalType(30, 2))
+ val (result, reader) = readAllWith(path, Seq("k", "d"),
+ createFilter(Seq(everyThirdRow), schema),
+ (b, i) => (b.column(0).getLong(i), b.column(1).getDecimal(i, 30, 2).toJavaBigDecimal))
+ try {
+ val expected = (0L until 1000L by 3L)
+ .map(k => (k, new java.math.BigDecimal(k).multiply(new java.math.BigDecimal("1.25"))
+ .setScale(2)))
+ assert(result == expected, s"got ${result.take(5)} (${result.size} rows)")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the survivor queue and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ val fileSchema = kvSchema()
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAll(path, filter, capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // Each survivor costs 9 bytes of long key and 40 of its own range. The sum passes the cap of
+ // 1024 at survivor 21, which releases the buffer, and the ranges alone at survivor 26, which
+ // gives the filter up. The capacity decides what the buffer holds when it is released. At
+ // 4096 that is one partly filled set of accumulators, and at 16 a full set in the queue as
+ // well. Both have to end with the queue empty and the filter given up. Keeping one without
+ // the other would leave phase 2 reading non-key columns while emit spliced keys from a queue
+ // holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAll(path, filter)
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSeq == Seq("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does, since the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers conjuncts under the relation's own column names") {
+ // The analyzer spells a reference the way the query did, so under case-insensitive resolution
+ // a bloom can name `_TMP_METADATA_ROW_INDEX` while the relation, and the schema the reader
+ // requests, call the column `_tmp_metadata_row_index`. The planner normalizes what it offers to
+ // the relation's names, so the format's gate sees the name the reader will see and declines.
+ // Without that the gate passed it, `ParquetStorageFilter.create` rejected it on the driver, and
+ // a query that runs with the conf off failed with it on.
+ withTempDir { dir =>
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", s"id AS $rowIndexName", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", rowIndexName, "v")
+ .queryExecution.optimizedPlan
+ def column(name: String): Attribute =
+ relation.output.find(_.name == name).getOrElse(fail(s"no $name in the relation output"))
+
+ def offered(key: Attribute): Seq[Expression] = {
+ val logical = LogicalFilter(
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ scanOf(physical).storageFilters
+ }
+
+ // As a control, an ordinary key spelled in another case is offered, under its own name.
+ val upperK = offered(column("k").withName("K"))
+ assert(upperK.size == 1, s"a bloom over `K` must reach the scan; got $upperK")
+ assert(upperK.head.references.map(_.name).toSet == Set("k"),
+ s"and it must carry the relation's name for the column; got ${upperK.head}")
+
+ val upperRowIndex =
+ offered(column(rowIndexName).withName(rowIndexName.toUpperCase(Locale.ROOT)))
+ assert(upperRowIndex.isEmpty,
+ s"a bloom over the row-index column must not be offered in any case; got $upperRowIndex")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers a conjunct that appears twice only once") {
+ // Spelled in two cases, a bloom over `k` normalizes to one conjunct, which the scan then takes
+ // once, as `afterScanFilters` does.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+ def bloomOn(key: Attribute): Expression =
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key)))
+ val logical = LogicalFilter(And(bloomOn(k), bloomOn(k.withName("K"))), relation)
Review Comment:
Done. The test is gone, and the one above checks `offered(k, k)`, which still fails without the `ExpressionSet`.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3708 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // The native address of an off-heap vector's values, 0 once it is freed. Only for a fixed-width
+ // vector. A string or binary one keeps its bytes in a child and reports 0 even while live.
+ private def address(vector: Any): Long =
Review Comment:
Done, it is after the log-message constants.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3708 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // The native address of an off-heap vector's values, 0 once it is freed. Only for a fixed-width
+ // vector. A string or binary one keeps its bytes in a child and reports 0 even while live.
+ private def address(vector: Any): Long =
+ vector.asInstanceOf[OffHeapColumnVector].valuesNativeAddress()
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t @ (_: VirtualMachineError | _: LinkageError) =>
+ fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`, and returns the rows.
+ private def readLaterSet(path: String, cap: Long): Seq[(String, String)] = {
Review Comment:
Done, both are local to that test now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4097172647
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its "
+ + "filter kept in row group %d of %s, but the file was written without a page "
+ + "index. Set %s=false to read this file.",
+ finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(),
+ SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new UnsupportedFileReadException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it --
+ // which can make the row group's contribution negative, and that is the truth about it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
+ }
+ }
+ }
+ remaining -= num;
+ }
+
+ if (accumulate) {
+ finalizePartialAccumulators();
+ }
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ // Assigned before the loop on purpose: an allocation failure part way through then leaves the
+ // vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows
+ * each.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, pushing them onto their
+ * queues once full. All key columns advance in lockstep, which is what keeps the queues aligned.
+ *
+ * <p>Returns false when the buffered survivors have passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} and this row group has
+ * given splicing up, in which case it has already released what it held and the caller must stop
+ * calling this.
+ */
+ private boolean appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ long valueBytes = 0L;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ // Measured on the destination: a dictionary-encoded source has no length of its own, since
+ // its values are read through the dictionary.
+ if (keyVariableLength[i]) valueBytes += dst.getArrayLength(dstRow);
+ }
+ }
+ splicedBytes += keyFixedBytesPerRow + valueBytes;
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ // Only here, so the cost is one comparison per capacity-sized vector rather than per row. A
+ // row group whose survivors fit in a single accumulator is never checked at all: it then
+ // holds one capacity-sized vector per key column, which is what a plain read holds anyway.
+ if (splicedBytes > storageFilter.maxSplicedRowGroupBytes()) {
+ abandonSplicing();
+ return false;
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+ }
+ return true;
+ }
+
+ /**
+ * Gives up splicing for the row group being evaluated and releases every survivor vector it has
+ * buffered. Phase 2 then reads the full projected schema and the emit path takes the persistent
+ * batch, so the rows are unaffected.
+ */
+ private void abandonSplicing() {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ closeAll(currentKeyAccumulators);
+ Arrays.fill(currentKeyAccumulators, null);
+ currentKeyAccumulatorRowCount = 0;
+ spliceCurrentRowGroup = false;
+ }
+
+ /**
+ * Pushes any partially-filled accumulator into its queue at row-group end so the emit path can
+ * dequeue it as the row group's final batch.
+ */
+ private void finalizePartialAccumulators() {
+ if (currentKeyAccumulatorRowCount == 0) return;
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Closes anything held by the splicing path: every vector still queued, the published head
+ * included, and partially-filled accumulators. Called from {@link #close()}.
+ */
+ private void closeSplicingState() {
+ keyVectorsPublished = false;
+ if (keyVectorQueues != null) {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ if (q != null) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ }
+ }
+ closeAll(currentKeyAccumulators);
+ currentKeyAccumulators = null;
+ }
+
+ /** Closes every non-null vector of {@code vectors}; tolerates a null array. */
+ private static void closeAll(WritableColumnVector[] vectors) {
+ if (vectors == null) return;
+ for (WritableColumnVector v : vectors) {
+ if (v != null) v.close();
+ }
+ }
+
+ /**
+ * Copies one key value between column vectors. Picked per key column at init time by
+ * {@link #copierFor(DataType)}; the caller handles null sources.
+ */
+ @FunctionalInterface
+ private interface ValueCopier {
+ void copy(WritableColumnVector dst, int dstRow, WritableColumnVector src, int srcRow);
+ }
+
+ /**
+ * Returns a {@link ValueCopier} for the given key {@link DataType}. The set of types handled here
+ * is the definition behind {@code ParquetStorageFilter.isSupportedKeyType}, which gates both
+ * planning-time extraction and {@code ParquetStorageFilter.create} -- so the throw at the end is
+ * unreachable. Teach both sides at once when adding a type; a type admitted there but missing
+ * here becomes a task failure instead of a planning-time rejection.
+ */
+ private static ValueCopier copierFor(DataType dt) {
Review Comment:
Kept as it is, for the reason you name as the condition. It is a vector-to-vector copy with no row in between, which is the whole point of it in that loop, and it only has to serve the types `isSupportedKeyType` admits.
The two lists cannot drift silently: a test asserts that `isSupportedKeyType` covers exactly the types `copierFor` handles, and the javadoc says to teach both sides at once.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its "
+ + "filter kept in row group %d of %s, but the file was written without a page "
+ + "index. Set %s=false to read this file.",
+ finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(),
+ SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new UnsupportedFileReadException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it --
+ // which can make the row group's contribution negative, and that is the truth about it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
+ }
+ }
+ }
+ remaining -= num;
+ }
+
+ if (accumulate) {
+ finalizePartialAccumulators();
+ }
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ // Assigned before the loop on purpose: an allocation failure part way through then leaves the
+ // vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows
+ * each.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, pushing them onto their
+ * queues once full. All key columns advance in lockstep, which is what keeps the queues aligned.
+ *
+ * <p>Returns false when the buffered survivors have passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} and this row group has
+ * given splicing up, in which case it has already released what it held and the caller must stop
+ * calling this.
+ */
+ private boolean appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ long valueBytes = 0L;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ // Measured on the destination: a dictionary-encoded source has no length of its own, since
+ // its values are read through the dictionary.
+ if (keyVariableLength[i]) valueBytes += dst.getArrayLength(dstRow);
+ }
+ }
+ splicedBytes += keyFixedBytesPerRow + valueBytes;
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ // Only here, so the cost is one comparison per capacity-sized vector rather than per row. A
+ // row group whose survivors fit in a single accumulator is never checked at all: it then
+ // holds one capacity-sized vector per key column, which is what a plain read holds anyway.
+ if (splicedBytes > storageFilter.maxSplicedRowGroupBytes()) {
+ abandonSplicing();
+ return false;
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+ }
+ return true;
+ }
+
+ /**
+ * Gives up splicing for the row group being evaluated and releases every survivor vector it has
+ * buffered. Phase 2 then reads the full projected schema and the emit path takes the persistent
+ * batch, so the rows are unaffected.
+ */
+ private void abandonSplicing() {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ closeAll(currentKeyAccumulators);
+ Arrays.fill(currentKeyAccumulators, null);
+ currentKeyAccumulatorRowCount = 0;
+ spliceCurrentRowGroup = false;
+ }
+
+ /**
+ * Pushes any partially-filled accumulator into its queue at row-group end so the emit path can
+ * dequeue it as the row group's final batch.
+ */
+ private void finalizePartialAccumulators() {
+ if (currentKeyAccumulatorRowCount == 0) return;
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Closes anything held by the splicing path: every vector still queued, the published head
+ * included, and partially-filled accumulators. Called from {@link #close()}.
+ */
+ private void closeSplicingState() {
+ keyVectorsPublished = false;
+ if (keyVectorQueues != null) {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ if (q != null) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ }
+ }
+ closeAll(currentKeyAccumulators);
+ currentKeyAccumulators = null;
+ }
+
+ /** Closes every non-null vector of {@code vectors}; tolerates a null array. */
+ private static void closeAll(WritableColumnVector[] vectors) {
+ if (vectors == null) return;
+ for (WritableColumnVector v : vectors) {
+ if (v != null) v.close();
+ }
+ }
+
+ /**
+ * Copies one key value between column vectors. Picked per key column at init time by
+ * {@link #copierFor(DataType)}; the caller handles null sources.
+ */
+ @FunctionalInterface
+ private interface ValueCopier {
+ void copy(WritableColumnVector dst, int dstRow, WritableColumnVector src, int srcRow);
+ }
+
+ /**
+ * Returns a {@link ValueCopier} for the given key {@link DataType}. The set of types handled here
+ * is the definition behind {@code ParquetStorageFilter.isSupportedKeyType}, which gates both
+ * planning-time extraction and {@code ParquetStorageFilter.create} -- so the throw at the end is
+ * unreachable. Teach both sides at once when adding a type; a type admitted there but missing
+ * here becomes a task failure instead of a planning-time rejection.
+ */
+ private static ValueCopier copierFor(DataType dt) {
+ if (dt instanceof BooleanType) {
+ return (dst, dRow, src, sRow) -> dst.putBoolean(dRow, src.getBoolean(sRow));
+ }
+ if (dt instanceof ByteType) {
+ return (dst, dRow, src, sRow) -> dst.putByte(dRow, src.getByte(sRow));
+ }
+ if (dt instanceof ShortType) {
+ return (dst, dRow, src, sRow) -> dst.putShort(dRow, src.getShort(sRow));
+ }
+ if (dt instanceof IntegerType
+ || dt instanceof DateType
+ || dt instanceof YearMonthIntervalType) {
+ return (dst, dRow, src, sRow) -> dst.putInt(dRow, src.getInt(sRow));
+ }
+ if (dt instanceof LongType
+ || dt instanceof TimestampType
+ || dt instanceof TimestampNTZType
+ || dt instanceof TimeType
+ || dt instanceof DayTimeIntervalType) {
+ return (dst, dRow, src, sRow) -> dst.putLong(dRow, src.getLong(sRow));
+ }
+ if (dt instanceof FloatType) {
+ return (dst, dRow, src, sRow) -> dst.putFloat(dRow, src.getFloat(sRow));
+ }
+ if (dt instanceof DoubleType) {
+ return (dst, dRow, src, sRow) -> dst.putDouble(dRow, src.getDouble(sRow));
+ }
+ if (dt instanceof DecimalType decimalType) {
+ int precision = decimalType.precision();
+ if (precision <= Decimal.MAX_INT_DIGITS()) {
+ return (dst, dRow, src, sRow) -> dst.putInt(dRow, src.getInt(sRow));
+ }
+ if (precision <= Decimal.MAX_LONG_DIGITS()) {
+ return (dst, dRow, src, sRow) -> dst.putLong(dRow, src.getLong(sRow));
+ }
+ return (dst, dRow, src, sRow) -> dst.putByteArray(dRow, src.getBinary(sRow));
+ }
+ // StringType covers CHAR and VARCHAR: both extend it.
+ if (dt instanceof StringType || dt instanceof BinaryType) {
+ return (dst, dRow, src, sRow) -> dst.putByteArray(dRow, src.getBinary(sRow));
Review Comment:
Follow-up in the description. The obvious shortcut does not work today: `OffHeapColumnVector.getByteBuffer` allocates a `byte[]` of its own, a dictionary-encoded source has to decode, and the two vector implementations disagree on whether the buffer position is absolute. Removing the copy needs the core column-vector classes, so it belongs in its own change.
Your point about plain-encoded high-cardinality keys is exactly right. The key-type tests now cover both encodings for real, with an assertion on what parquet actually wrote, which turned out to matter: the previous dictionary arm was writing PLAIN files.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Literal, Predicate}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.{BinaryType, BooleanType, ByteType, DataType, DateType, DayTimeIntervalType, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType, StringType, StructType, TimestampNTZType, TimestampType, TimeType, YearMonthIntervalType}
+
+/**
+ * Optional SQL metrics the reader updates while applying a [[ParquetStorageFilter]]. Every
+ * counter is scoped to what the storage filter added on top of a read of the same projection
+ * without one. All fields are nullable; a null field disables that metric.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums the rows those skips excluded, per skipped block the rows that
+ * survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it, and it is zero on an
+ * all-keys projection, which can avoid nothing.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned.
+ *
+ * The row counters' suffix says where a row was excluded, not what would have saved it: a row
+ * inside a kept row group is read as part of its page and dropped during decode, so page
+ * filtering did not save it. An all-keys projection has no page filtering at all, and
+ * [[rowsExcludedWithinRowGroup]] still counts every row the filter dropped.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric = null,
+ rowsExcludedByRowGroup: SQLMetric = null,
+ rowsExcludedWithinRowGroup: SQLMetric = null,
+ bytesAvoidedByRowGroup: SQLMetric = null,
+ bytesAvoidedByPageFiltering: SQLMetric = null)
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[boundExpression]] has its references rewritten to
+ * [[BoundReference]]s pointing at positions 0..(keyColumnIndices.length - 1); the reader must
+ * evaluate it against rows whose fields correspond to those key columns in that order.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ val boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer
+ // construction to first use on the executor.
+ @transient private lazy val predicate: BasePredicate = Predicate.create(boundExpression)
+
+ def test(keyRow: InternalRow): Boolean = predicate.eval(keyRow)
+
+ /**
+ * Returns a new filter for a physical file that is missing some key columns (schema evolution).
+ * The [[BoundReference]]s at `missingKeyLocalPositions`, which are indices into
+ * [[keyColumnIndices]], are replaced by `missingKeyValues`, and the remaining references are
+ * renumbered onto the reduced key-row layout. [[keyColumnIndices]] keeps the present columns in
+ * their original relative order, and SQL metrics are shared with `this`.
+ *
+ * `missingKeyValues(i)` must be the internal-format value the reader produces for a missing
+ * column: its existence DEFAULT when it has one, else null. `ParquetColumnVector` writes that
+ * default into the output vector, so substituting null instead would filter on a value the scan
+ * never returns and could drop matching rows.
+ *
+ * The predicate has to be evaluated against the substitution rather than skipped, because a null
+ * key does not always mean `false`: a `Coalesce`-wrapped reference still produces a non-null
+ * result, and `XxHash64` is `nullable = false` and hashes a null input to its seed.
+ *
+ * With every key position missing, the result holds no [[BoundReference]] at all and
+ * [[evalAllMissing]] can read off its constant truth value.
+ */
+ def rewriteForMissingKeys(
+ missingKeyLocalPositions: Array[Int],
+ missingKeyValues: Array[Any]): ParquetStorageFilter = {
+ require(missingKeyLocalPositions.length == missingKeyValues.length,
+ "missingKeyLocalPositions and missingKeyValues must have the same length")
+ val substitution = missingKeyLocalPositions.zip(missingKeyValues).toMap
+ val presentPositions = keyColumnIndices.indices.filterNot(substitution.contains)
+ val newPosOf = presentPositions.zipWithIndex.toMap
+ val rewritten = boundExpression.transform {
Review Comment:
Fixed: the rewrite is memoized per set of missing key positions for the task, so the bloom is deserialized once per scan rather than once per file. The substituted values are a function of those positions, since they come from the scan's schema, which is why the positions alone are a sound key. c255f88
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -409,23 +598,276 @@ public boolean nextBatch() throws IOException {
}
cv.assemble();
}
- // If needed, compute row indexes within a file.
- if (rowIndexGenerator != null) {
- rowIndexGenerator.populateRowIndex(columnVectors, num);
- }
+ }
- rowsReturned += num;
+ /** Publishes {@code num} rows as the current batch. */
+ private void finishBatch(int num) {
columnarBatch.setNumRows(num);
+ rowsReturned += num;
numBatched = num;
batchIdx = 0;
+ }
+
+ /**
+ * Splicing emit path. Publishes one survivor key vector per key column into the batch, reads the
+ * non-key slots for {@code num} rows, and hands out the same {@link ColumnarBatch} every time,
+ * its key slots rewritten in place. The batch is a view over vectors owned elsewhere; see
+ * {@link #close()} and {@link #closeSplicingState()}.
+ */
+ private boolean nextBatchSplicing() throws IOException {
+ releasePublishedKeyVectors();
+ for (ParquetColumnVector cv : columnVectors) {
+ cv.reset();
+ }
+ // Zero the outgoing batch before the terminal checks below, as the plain path does. Otherwise a
+ // terminal call leaves the batch pointing at key vectors that were just closed -- off-heap,
+ // that is freed memory.
+ if (columnarBatch != null) columnarBatch.setNumRows(0);
+ if (hitEndOfData) return false;
+ if (rowsReturned >= totalRowCount) return false;
+ checkEndOfRowGroup();
+ if (hitEndOfData) return false;
+
+ int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned);
+
+ if (!spliceCurrentRowGroup) {
+ // This row group was read the plain way: phase 2 took every projected column, so every slot
+ // of the batch comes from the persistent one.
+ readPersistentColumns(num, /* skipKeySlots= */ false);
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.populateRowIndex(columnVectors, num);
+ }
+ System.arraycopy(persistentBatchColumns, 0, spliceBatchColumns, 0, spliceBatchColumns.length);
+ finishBatch(num);
+ return true;
+ }
+
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ if (keyVectorQueues[i].isEmpty()) {
+ // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop
+ // is driven by that same count. Named rather than left to NoSuchElementException because
+ // this one must not be swallowed: under ignoreCorruptFiles that would silently drop the
+ // rest of a healthy file's rows.
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to "
+ + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num));
+ }
+ }
+ // The queues keep owning these until the next emit releases them, so a phase-2 read below that
+ // throws leaves them reachable for `close()`.
+ keyVectorsPublished = true;
+ // Key slots are filled in ascending slot order while `keyIdx` walks the queues in key-list
+ // order, so the pairing is the identity only because `ParquetStorageFilter.create` sorts
+ // `keyColumnIndices` ascending. `isKeyTopLevel` says which slots are keys, not where each
+ // sits in that list, so this loop cannot re-derive the pairing: an unsorted list would swap
+ // key columns in the output batch.
+ int keyIdx = 0;
+ for (int i = 0; i < spliceBatchColumns.length; i++) {
+ if (i < isKeyTopLevel.length && isKeyTopLevel[i]) {
+ spliceBatchColumns[i] = keyVectorQueues[keyIdx++].peekFirst();
+ } else {
+ spliceBatchColumns[i] = persistentBatchColumns[i];
+ }
+ }
+
+ readPersistentColumns(num, /* skipKeySlots= */ true);
+ if (rowIndexGenerator != null) {
+ // Row-index column is identified by name (ROW_INDEX_TEMPORARY_COLUMN_NAME), which is a
+ // synthetic metadata column never referenced by a storage filter, so its slot is a non-key
+ // slot with a persistent ParquetColumnVector.
+ rowIndexGenerator.populateRowIndex(columnVectors, num);
+ }
+ finishBatch(num);
return true;
}
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (java.util.ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
private void initializeInternal() throws IOException, UnsupportedOperationException {
missingColumns = new HashSet<>();
for (ParquetColumn column : CollectionConverters.asJava(parquetColumn.children())) {
checkColumn(column);
}
+ if (storageFilter != null) {
+ initializeLateMaterialization();
+ }
+ }
+
+ /**
+ * Sets the storage filter for late materialization. Must be called before {@link #initialize};
+ * {@link #initializeLateMaterialization()} then inspects the per-file schema and decides whether
+ * splicing engages.
+ *
+ * <p>It does not engage in one real case: every key column is missing from this physical file
+ * under schema evolution. The predicate is then rewritten with each missing key replaced by the
+ * constant the reader materializes for it (its existence DEFAULT, else null) and evaluated once
+ * -- true keeps the file unfiltered, false or null skips it.
+ *
+ * <p>Every other precondition is guaranteed by
+ * {@code FileSourceStrategy.extractStorageFilters} and {@code ParquetStorageFilter.create}, and a
+ * violation throws rather than falling back: extraction has already removed the filter from the
+ * post-scan Filter, so not applying it would return rows the query rejected.
+ */
+ public void setStorageFilter(ParquetStorageFilter storageFilter) {
+ this.storageFilter = storageFilter;
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ lateMatReader = reader.getUnderlyingReader();
+ if (lateMatReader == null) {
+ // Unreachable: the only ParquetRowGroupReader ParquetFileFormat builds is
+ // ParquetRowGroupReaderImpl, which exposes its reader.
+ throw new UnsupportedFileReadException(
+ "Storage-filter pushdown requires a reader backed by a ParquetFileReader, but "
+ + reader.getClass().getName() + " does not expose one");
+ }
+ if (configuration != null) {
+ useColumnIndexFilter = configuration.getBoolean(
+ ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ }
+
+ // Resolve each key column's top-level ParquetColumn. Partition into present and missing (the
+ // latter can happen under schema evolution: a column is in the requested schema but not in this
+ // physical parquet file). For any non-primitive key we still bail; phase-1 reads only primitive
+ // leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> missingKeyLocalPositions = new ArrayList<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new UnsupportedFileReadException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (missingColumns.contains(column)) {
+ missingKeyLocalPositions.add(i);
+ } else {
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // If any key column is missing from this file, rewrite the predicate to substitute the constant
+ // the reader will actually materialize for that column. That is the column's existence DEFAULT
+ // when it has one (ParquetColumnVector writes it into the output vector and marks the vector
+ // constant), otherwise null. Substituting null for a column that reads back as its default
+ // would filter on a value the scan never returns. The predicate must be evaluated against the
+ // substituted constant rather than skipped: null does not always mean false in a filter.
+ if (!missingKeyLocalPositions.isEmpty()) {
+ int[] missing = new int[missingKeyLocalPositions.size()];
+ Object[] missingValues = new Object[missingKeyLocalPositions.size()];
+ Object[] existenceDefaults =
+ ResolveDefaultColumns.existenceDefaultValues(sparkRequestedSchema);
+ for (int i = 0; i < missing.length; i++) {
+ missing[i] = missingKeyLocalPositions.get(i);
+ missingValues[i] = existenceDefaults[keyIndices[missing[i]]];
+ }
+ storageFilter = storageFilter.rewriteForMissingKeys(missing, missingValues);
+
+ if (presentKeyColumns.isEmpty()) {
Review Comment:
Fixed: the skip now walks every row group of the file and updates all three counters.
It measures against the rows the pushed data filter kept, which is the baseline every other skip path uses. Measuring against the whole row group would have let `rowsExcludedByRowGroup` exceed the reader's own row count, since that is `getFilteredRecordCount()`. A test asserts the numbers. c255f88
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala:
##########
@@ -165,6 +166,57 @@ trait FileFormat {
}
}
+ /**
+ * Like [[buildReaderWithPartitionValues]] but additionally accepts a sequence of storage filters:
+ * Catalyst expressions that the storage layer may evaluate to drive value-column IO pruning based
+ * on key-column evaluation (e.g., late materialization with a runtime bloom filter).
+ *
+ * The default implementation delegates to [[buildReaderWithPartitionValues]] and accepts no
+ * storage filter at all. A format that supports storage-filter pushdown overrides this, and must
+ * not let the two builders call each other: this default delegates one way, so an override that
+ * delegates back recurses until the driver's stack runs out, `super` included, since that call is
+ * virtual too. Route both to a private implementation instead, the way `ParquetFileFormat` does.
+ *
+ * A non-empty `storageFilters` here is a planner bug, so the default body rejects it rather than
+ * dropping it: the planner removes an extracted conjunct from the post-scan `Filter`, so a reader
+ * that ignores it returns rows the filter rejects.
+ *
+ * Scalar subqueries inside `storageFilters` are expected to have been materialized before this
+ * method is called, so that the returned reader can be safely serialized to executors.
+ *
+ * `storageFilterMetrics` is an optional map of SQL metrics the reader can update during execution
+ * (e.g. number of row groups skipped). The scan is expected to expose these metrics via its
+ * `metrics` field so they show up in the SQL UI.
+ */
+ def buildReaderWithStorageFilters(
+ sparkSession: SparkSession,
+ dataSchema: StructType,
+ partitionSchema: StructType,
+ requiredSchema: StructType,
+ filters: Seq[Filter],
+ storageFilters: Seq[Expression],
+ options: Map[String, String],
+ hadoopConf: Configuration,
+ storageFilterMetrics: Map[String, SQLMetric] = Map.empty
+ ): PartitionedFile => Iterator[InternalRow] = {
+ require(storageFilters.isEmpty,
Review Comment:
Resolved from the other end by the model change: honoring a storage filter is optional, so this default is a legitimate implementation rather than dead code. It ignores the filters and delegates, which is correct precisely because the conjunct is still in the `Filter`, and the `require` is gone.
The recursion warning stays, with a corrected reason. The default delegates on `this`, so an override of `buildReaderWithPartitionValues` that delegates back here closes the loop. Reaching this default through `super` is part of that loop rather than an escape from it, which is what the old wording got wrong. c255f88
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1885,6 +1885,42 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED =
Review Comment:
Done, both confs and their accessors are after `PARQUET_FILTER_PUSHDOWN_INFILTERTHRESHOLD` now, so the `filterPushdown` group stays with its sub-confs. c255f88
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4204523275
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3708 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // The native address of an off-heap vector's values, 0 once it is freed. Only for a fixed-width
+ // vector. A string or binary one keeps its bytes in a child and reports 0 even while live.
+ private def address(vector: Any): Long =
+ vector.asInstanceOf[OffHeapColumnVector].valuesNativeAddress()
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t @ (_: VirtualMachineError | _: LinkageError) =>
+ fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`, and returns the rows.
+ private def readLaterSet(path: String, cap: Long): Seq[(String, String)] = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString),
+ capacity = 4)
+ reader.close()
+ rows
+ }
+
+ test("a later accumulator set of a variable-length key is charged per value") {
+ // Six survivors over a capacity of 4 fill one set of four 1000-byte keys and a second of two
+ // 2000-byte ones. Every set grows as its values arrive, the way the first one does. Each value
+ // is charged four times its bytes plus its row's fixed part of 17, and the one range adds 40.
+ // So the row group is charged 4 * (4000 + 17) + 2 * (8000 + 17) + 40 = 32142 bytes. A cap of
+ // 32142 keeps splicing and one of 32141 gives it up, which a missing or a double charge would
+ // each get wrong. And the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ assertSpliceCharge(32142L, rows = 6) { cap =>
+ readUnderCap(path, laterSetSchema, laterSetFilter, cap, batchSize = 4)
+ }
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val rows = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ }
+ }
+ }
+
+ test("what a row group of scattered survivors is charged does not depend on the cap") {
+ // Keys 9 to 12 fill the first set over a capacity of 4, and keys 14 and 17 come scattered, each
+ // in a range of its own. Every set grows as values arrive and is charged per value, so the row
+ // group's charge is one number: the first set's 16068, three ranges of 40 and the second
+ // set's 8034, 24222 in all. A cap of it splices and one byte under it gives splicing up.
+ // Larger caps splice too, which a few of them check, among them 32216 to 32221. Those gave
+ // splicing up when a later set was sized from the one before it with no room for its ranges.
+ // The second set is not sized at all, so its byte child grows to 2000 bytes for its two
+ // 1000-byte values, where sizing it from the first set would have reserved 8000.
+ withTempDir { dir =>
+ val (path, _) = writeLaterSetFile(dir, lastTwoBytes = 1000)
+ val prefix = Substring(BoundReference(0, StringType, nullable = true), Literal(1), Literal(4))
+ val keys = Seq("0009", "0010", "0011", "0012", "0014", "0017").map(Literal(_))
+ def readUnder(cap: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, Seq(In(prefix, keys)), cap, batchSize = 4)
+ assertSpliceCharge(24222L, rows = 6)(readUnder)
+ val (_, uncapped) = readUnder("64MB")
+ Seq(32215L, 32216L, 32221L, 32222L).foreach { cap =>
+ val (rows, bytes) = readUnder(s"${cap}b")
+ assert(rows == 6 && bytes == uncapped,
+ s"a cap of $cap must keep the six rows and splicing; got $rows rows")
+ }
+ val (children, reader) = readAllWith(path, Seq("k", "v"),
+ createFilter(Seq(In(prefix, keys)), laterSetSchema),
+ (b, _) => ParquetTestAccess.byteChildCapacity(
+ b.column(0).asInstanceOf[WritableColumnVector]),
+ capacity = 4)
+ reader.close()
+ assert(children.drop(4).distinct == Seq(2000),
+ s"the second set's byte child must grow as values arrive; got $children")
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ assertSpliceCharge(166L, rows = 7) { cap =>
+ readUnderCap(path, schema, storageFilters, cap, batchSize = 16)
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warnings) = withLogged(partialReadWarning) {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(warnings == 0,
+ s"the footer check must get there before the exception path; got $warnings")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warnings) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(warnings == 0, "the footer check must decline the row group rather than the read " +
+ s"discovering it; got $warnings")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize = 64L * 1024 * 1024,
+ options = Map(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT -> "100"))
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ // The rows a reader hands back before it throws, what it throws, and, for the filtering
+ // reader, whether its scratch vectors were replaced on the way and the old ones freed. Both
+ // read off heap, where an old scratch vector left open would leak native memory.
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable, Boolean) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(true, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(true, 16, storageFilter)
+ }
+ def scratch: Seq[WritableColumnVector] = reader match {
+ case filtering: LateMaterializationParquetRecordReader =>
+ ParquetTestAccess.keyScratchVectors(filtering)
+ case _ => Nil
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val scratchBefore = scratch
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ val replacedAndFreed = scratch.zip(scratchBefore).exists { case (now, before) =>
+ (now ne before) && address(before) == 0
+ }
+ (read.toSeq, error, replacedAndFreed)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError, _) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError, scratchReplaced) =
+ readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ // A failed decode can leave the scratch vectors grown. Phase 2 reads the same pages into
+ // vectors of its own, so the reader replaces them with capacity-sized ones rather than keep
+ // the grown ones for the rest of the split.
+ assert(scratchReplaced,
+ "the decoding fallback must replace the scratch vectors and free the old ones")
+
+ // A kill that shows only at the decoding catch, after the per-chunk check has passed, is
+ // seen there. Without that catch's own check the row group would fall back and phase 2
+ // would fail on the page instead.
+ val killedAtTheCatch = spy(TaskContext.empty())
+ doAnswer((_: InvocationOnMock) => Boolean.box(Thread.currentThread().getStackTrace
+ .exists(_.getMethodName.startsWith("rethrowIfMustPropagate"))))
+ .when(killedAtTheCatch).isInterrupted()
+ val (_, killed, _) = TaskContext.withTaskContext(killedAtTheCatch) {
+ readUntilFailure(keyAtLeastFilter(500L))
+ }
+ assert(killed.isInstanceOf[SparkException] && killed.getMessage.contains("task kill") &&
+ killed.getCause != null && killed.getCause.getClass == plainError.getClass,
+ s"the kill must go out from the decoding catch, around the decoding error; got $killed")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+
+ test("a key vector that cannot grow, or a codec's InternalError, gives the row group up") {
+ // Both are fatal-looking errors that decoding the key pages can raise, and the reader falls
+ // back on both. A scratch vector that cannot grow is phase 1's own, and phase 2 then reads
+ // what a plain read reads. An `InternalError` from decoding is how a codec like hadoop-lzo
+ // reports a corrupt page, which the plain read of the same pages meets again. Raised from the
+ // filter instead, the first goes out as its OutOfMemoryError and the second in an
+ // `UnknownError`. Off heap, the scratch vectors the failed decode used must be freed, and the
+ // second row group's phase 1 runs over the ones that replace them.
+ withTempDir { dir =>
+ val rows = (0L until 40L).map(i => (i, i.toString))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "20"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(20L, 20L),
+ "the fixture must be two row groups of 20 rows")
+ Seq(
+ new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve")),
+ new InternalError("codec")).foreach { error =>
+ var failedScratch: Seq[WritableColumnVector] = Nil
+ // Only the first row group's key reader fails.
+ val failing: KeyReaderHook.OnKeyReader = (reader, keyReader) => {
+ val scratch = ParquetTestAccess.keyScratchVectors(
+ reader.asInstanceOf[LateMaterializationParquetRecordReader])
+ if (failedScratch.nonEmpty) {
+ // Checked before phase 1 writes into them, since a freed one would crash the JVM.
+ assert(scratch.forall(v => !failedScratch.exists(_ eq v) && address(v) != 0),
+ "the second row group must decode into new scratch vectors")
+ keyReader
+ } else {
+ failedScratch = scratch
+ val failingReader = spy(keyReader)
+ doThrow(error).when(failingReader).readBatch(anyInt(), any(), any(), any())
+ failingReader
+ }
+ }
+ val ((result, reader), warnings) = try {
+ withLogged(decodingGivenUp) {
+ readAll(path, keyAtLeastFilter(30L), useOffHeap = true, onKeyReader = failing)
+ }
+ } catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case e: VirtualMachineError => fail(s"the reader must fall back on $error; got $e", e)
+ }
+ try {
+ assert(result == rows.take(20) ++ rows.drop(30),
+ s"the first row group must come back whole and the second filtered on $error; got " +
+ s"${result.map(_._1)}")
+ assert(warnings == 1, s"the decoding fallback must report $error; got $warnings")
+ assert(failedScratch.forall(v => address(v) == 0),
+ s"the scratch vectors of the failed decode must be freed on $error")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("a key value the filter does not read and cannot be decoded gives splicing up") {
+ // The filter need not read every key on every row, as `coalesce(a, b)` reads `b` only where
+ // `a` is null. The copy into the survivor buffer reads every key, so it can meet a value no
+ // plain read decodes. A dictionary-encoded column decodes a value only when it is read, so a
+ // dictionary that holds fewer values than its ids point at is such a value. Copying it gives
+ // splicing up for the row group, and phase 2 reads the key columns the way a plain read does.
+ // So the read answers where a plain one does, and the filter still applies.
+ withTempDir { dir =>
+ // Two copies of each `b`, so parquet keeps its dictionary. Cut to 64 values, it leaves `b`
+ // undecodable from row 128 on. `a` is null only before that, so the filter never reads a
+ // damaged `b`.
+ val rows = (0 until 200).map { i =>
+ (if (i < 100 && i % 5 == 0) null else s"a_$i", s"b_${i / 2}", i)
+ }
+ val path = writeSingleParquetFile(dir, rows.toDF("a", "b", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE, dictionary = true)
+ assert(encodingsOf(path, "b").exists(_.usesDictionary),
+ s"the fixture's whole point; got ${encodingsOf(path, "b")}")
+ shrinkDictionary(path, column = "b", entries = 64)
+ val columns = Seq("a", "b", "v")
+ intercept[ArrayIndexOutOfBoundsException] {
+ readAllWith(path, columns, null, (b, i) => b.column(1).getUTF8String(i).toString)._2
+ .close()
+ }
+
+ val schema = StructType(Seq(
+ StructField("a", StringType), StructField("b", StringType), StructField("v", IntegerType)))
+ val coalesced = Coalesce(Seq(BoundReference(0, StringType, nullable = true),
+ BoundReference(1, StringType, nullable = true)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(coalesced, Literal.create("a_15", StringType))), schema)
+ val ((result, reader), givenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, columns, filter, (b, i) => b.column(2).getInt(i), capacity = 16)
+ }
+ try {
+ // Keeps rows 150-199 among others, whose `b` is damaged.
+ val expected = rows.filter { case (a, b, _) => Option(a).getOrElse(b) >= "a_15" }.map(_._3)
+ assert(expected.contains(150), "the filter must keep a row whose `b` is damaged")
+ assert(result == expected, s"got ${result.size} rows, expected ${expected.size}")
+ assert(givenUp == 1,
+ "the copy must have failed and given splicing up once, or this proves nothing; " +
+ s"logged $givenUp")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry reads the damaged row group whole over the pushed data filter's ranges, and only
+ // that one, since parquet keeps a column index store per block. What it must not give up is
+ // the pushed data filter's own page pruning, which a plain read of the same query keeps, nor
+ // the other row groups' narrowing. Dropping either would read more than it needs to.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warnings) = withLogged(partialReadWarning)(read(evenKeys))
+ assert(warnings == 1,
+ "the read must have discovered the damaged index, or this test proves nothing; got " +
+ warnings)
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
Review Comment:
Thanks, done. The test compares the exact ordered output against that, which also replaces the size and membership checks it covers.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1032 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A fixed-width key needs no
+ * factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. It is part of `keyFixedBytesPerRow`, counted
+ * twice, for its data byte and its null byte.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // decoders of their last page.
+ closeDataPages();
+ releaseRowGroupReaders();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
Review Comment:
Thanks. I'd keep this one for a follow-up, as you suggest for further optimizations. One batch of the longest keys bounds it, so it needs a threshold to be worth the reallocation. It is in the description's follow-up list now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-6036943224
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/0c14861e323174452d42ffb7947d1c1158c39286
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth closed pull request #58895: [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader
URL: https://github.com/apache/spark/pull/58895
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5972272279
@dongjoon-hyun thank you for round 9. It is answered in the threads, and the description is up to date.
The three you put first are fixed:
- A decoding fallback replaces the key scratch vectors, so phase 2 allocates what a plain read does.
- An `InternalError` from the filter goes out in an `UnknownError`, so a pending kill no longer keeps the executor running, and the depth caveat is gone.
- All three catches go through one `giveUpOnError` again, and a test pins the decoding catch's rethrow.
The benchmark is reworked for R9-5 and R9-8. Every case also runs as a scan alone, and the cases are now 0.1% to 5% on a wide and a narrow table, each in small and default pages. The description has the JDK 17 numbers. The wide table in small pages gains up to 5.0x on the scan alone, and the narrow table with scattered keys costs up to 20%.
I'd keep R9-7 as it is, with the reason in its thread. R9-10 is in the follow-ups.
No round since the third has found a wrong answer, and none under the default settings, which keep the feature off. Is there anything you would see as blocking a merge? I'd like to handle what is left beyond that as follow-ups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5797805610
Thank you for the review, @dongjoon-hyun -- it was unusually thorough, and two of the comments changed the design rather than the code.
Every comment is addressed, in one commit on top of the reviewed one (b9981cda93b) so the delta stays readable. Replies are inline. The larger ones:
- The survivor buffer is bounded now, in the cheaper form you suggested: phase 1 counts the bytes it has buffered and gives that row group up past an internal cap, after which phase 2 reads every projected column of the surviving rows.
- `FileFormat.supportsStorageFilter` replaces the planner's exact-class check, so `FileSourceStrategy` no longer imports the parquet package.
- The offset-index check is gone. Phase 2's read is wrapped instead and parquet's own `MissingOffsetIndexException` rethrown with guidance, so a file without a page index still scans as long as the filter never prunes inside a row group.
- Every loud failure on this path now throws a new `UnsupportedFileReadException` that `ignoreCorruptFiles` does not swallow. Chasing your offset-index comment turned that up: `shouldIgnoreCorruptFileException` matches any `RuntimeException`, so before this every failure the feature depends on could have been read as a corrupt file and silently dropped the rest of a healthy one's rows.
- Your design question also produced a planner gate. An all-keys projection is no longer pushed at all: the reader has to read a key column to evaluate the filter on it, so such a scan reads the same columns for the same rows as a plain one and pushing can only add cost.
The description now carries a **Design decisions** section answering the design question up front, and a measured table on a real workload's file at the end of the test section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4072847200
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -421,11 +552,313 @@ public boolean nextBatch() throws IOException {
return true;
}
+ /**
+ * Splicing emit path. Closes the previous emit's dequeued key vectors (releasing survivor memory
+ * incrementally), advances to the next row group if needed via {@link #checkEndOfRowGroup()},
+ * dequeues one survivor key vector per key column, drives non-key column readers for {@code num}
+ * rows, and assembles a fresh {@link ColumnarBatch} interleaving key (dequeued) and non-key
+ * (persistent value-vector) slots in the original projection order.
+ * The per-emit batch is a transient view over vectors owned elsewhere; see {@link #close()} and
+ * {@link #closeSplicingState()}.
+ */
+ private boolean nextBatchSplicing() throws IOException {
+ if (pendingCloseKeyVectors != null) {
+ for (WritableColumnVector v : pendingCloseKeyVectors) {
+ if (v != null) v.close();
+ }
+ pendingCloseKeyVectors = null;
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) continue;
+ columnVectors[i].reset();
+ }
+ // Match the eager path and zero the outgoing batch before the terminal checks below. Without
+ // this, a terminal call leaves `columnarBatch` pointing at the previous emit whose key slots
+ // were just closed above -- with off-heap vectors those buffers are already freed, so a
+ // consumer that read the batch after nextBatch() returned false would see freed memory.
+ if (columnarBatch != null) columnarBatch.setNumRows(0);
+ if (hitEndOfData) return false;
+ if (rowsReturned >= totalRowCount) return false;
+ checkEndOfRowGroup();
+ if (hitEndOfData) return false;
+
+ int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned);
+
+ WritableColumnVector[] dequeued = new WritableColumnVector[keyVectorQueues.length];
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ dequeued[i] = keyVectorQueues[i].removeFirst();
Review Comment:
**Leak: these vectors are unreachable by `close()` until line 632.**
`removeFirst()` takes them out of `keyVectorQueues`, but `pendingCloseKeyVectors = dequeued` is only reached at the end of the method. If `columnReader.readBatch(...)` (L598) or `cv.assemble()` (L602) throws -- a corrupt page, an object-store read error -- the vectors are in neither the queues nor `pendingCloseKeyVectors`, so `close()` -> `closeSplicingState()` walks only the remaining queue entries and skips them.
With `spark.sql.columnVector.offheap.enabled=true` these are `Platform.allocateMemory` allocations freed only by `releaseMemory()` (no finalizer, no `Cleaner`), so each of the 4 task attempts leaks `numKeys * capacity` bytes permanently.
Moving `pendingCloseKeyVectors = dequeued;` to immediately after this loop closes it.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -421,11 +552,313 @@ public boolean nextBatch() throws IOException {
return true;
}
+ /**
+ * Splicing emit path. Closes the previous emit's dequeued key vectors (releasing survivor memory
+ * incrementally), advances to the next row group if needed via {@link #checkEndOfRowGroup()},
+ * dequeues one survivor key vector per key column, drives non-key column readers for {@code num}
+ * rows, and assembles a fresh {@link ColumnarBatch} interleaving key (dequeued) and non-key
+ * (persistent value-vector) slots in the original projection order.
+ * The per-emit batch is a transient view over vectors owned elsewhere; see {@link #close()} and
+ * {@link #closeSplicingState()}.
+ */
+ private boolean nextBatchSplicing() throws IOException {
+ if (pendingCloseKeyVectors != null) {
+ for (WritableColumnVector v : pendingCloseKeyVectors) {
+ if (v != null) v.close();
+ }
+ pendingCloseKeyVectors = null;
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) continue;
+ columnVectors[i].reset();
+ }
+ // Match the eager path and zero the outgoing batch before the terminal checks below. Without
+ // this, a terminal call leaves `columnarBatch` pointing at the previous emit whose key slots
+ // were just closed above -- with off-heap vectors those buffers are already freed, so a
+ // consumer that read the batch after nextBatch() returned false would see freed memory.
+ if (columnarBatch != null) columnarBatch.setNumRows(0);
+ if (hitEndOfData) return false;
+ if (rowsReturned >= totalRowCount) return false;
+ checkEndOfRowGroup();
+ if (hitEndOfData) return false;
+
+ int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned);
+
+ WritableColumnVector[] dequeued = new WritableColumnVector[keyVectorQueues.length];
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ dequeued[i] = keyVectorQueues[i].removeFirst();
+ }
+
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) continue;
+ ParquetColumnVector cv = columnVectors[i];
+ for (ParquetColumnVector leafCv : cv.getLeaves()) {
+ VectorizedColumnReader columnReader = leafCv.getColumnReader();
+ if (columnReader != null) {
+ columnReader.readBatch(num, leafCv.getValueVector(),
+ leafCv.getRepetitionLevelVector(), leafCv.getDefinitionLevelVector());
+ }
+ }
+ cv.assemble();
+ }
+ if (rowIndexGenerator != null) {
+ // Row-index column is identified by name (ROW_INDEX_TEMPORARY_COLUMN_NAME), which is a
+ // synthetic metadata column never referenced by a storage filter, so its slot is a non-key
+ // slot with a persistent ParquetColumnVector.
+ rowIndexGenerator.populateRowIndex(columnVectors, num);
+ }
+
+ ColumnVector[] cols = new ColumnVector[persistentBatchColumns.length];
+ // This walks batch slots in ascending order while `keyIdx` walks the survivor queues in
+ // key-row-position order, so it pairs the k-th smallest key slot with key-row position k.
+ // That is only the identity because `ParquetStorageFilter.create` sorts `keyColumnIndices`
+ // ascending -- see the comment there. `isKeyTopLevel` marks which slots are keys but not their
+ // position in that list, so this loop cannot reconstruct the pairing on its own: if the list
+ // ever stops being sorted, key columns silently swap places in the output batch.
+ int keyIdx = 0;
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ if (i < isKeyTopLevel.length && isKeyTopLevel[i]) {
+ cols[i] = dequeued[keyIdx++];
+ } else {
+ cols[i] = persistentBatchColumns[i];
+ }
+ }
+ columnarBatch = new ColumnarBatch(cols);
+ columnarBatch.setNumRows(num);
+
+ rowsReturned += num;
+ numBatched = num;
+ batchIdx = 0;
+ pendingCloseKeyVectors = dequeued;
+ return true;
+ }
+
private void initializeInternal() throws IOException, UnsupportedOperationException {
missingColumns = new HashSet<>();
for (ParquetColumn column : CollectionConverters.asJava(parquetColumn.children())) {
checkColumn(column);
}
+ if (storageFilter != null) {
+ initializeLateMaterialization();
+ }
+ }
+
+ /**
+ * Sets the storage filter for late materialization. Must be called before {@link #initialize};
+ * {@link #initializeLateMaterialization()} (run from {@link #initialize}) inspects the per-file
+ * schema and decides whether splicing actually engages.
+ *
+ * <p>Splicing does NOT engage in one real case: all key columns are missing from this physical
+ * file under schema evolution. The predicate is rewritten with each missing key replaced by the
+ * constant the reader materializes for it (its existence DEFAULT, else null -- see
+ * {@code ParquetStorageFilter.rewriteForMissingKeys}) and evaluated as a constant: a true result
+ * keeps the file with no filtering, a false/null result skips it entirely. Either way the rows
+ * the scan returns are exactly the rows that satisfy the filter, so this is safe.
+ *
+ * <p>Every OTHER precondition is guaranteed by planning-time checks in
+ * {@code FileSourceStrategy.extractStorageFilters} and {@code ParquetStorageFilter.create}, and
+ * violations throw rather than fall back: once extraction has moved a bloom filter onto the scan
+ * it is gone from the post-scan Filter, so quietly not applying it would produce wrong rows.
+ */
+ public void setStorageFilter(ParquetStorageFilter storageFilter) {
+ this.storageFilter = storageFilter;
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ lateMatReader = reader.getUnderlyingReader();
+ if (lateMatReader == null) {
+ // Unreachable in production: the only ParquetRowGroupReader ParquetFileFormat builds is
+ // ParquetRowGroupReaderImpl, which returns its reader. Dropping the filter here would return
+ // rows it rejects, since extraction already removed it from the post-scan Filter.
+ throw new IllegalStateException(
+ "Storage-filter pushdown requires a reader backed by a ParquetFileReader, but "
+ + reader.getClass().getName() + " does not expose one");
+ }
+ if (configuration != null) {
+ useColumnIndexFilter = configuration.getBoolean(
+ ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ }
+
+ // Resolve each key column's top-level ParquetColumn. Partition into present and missing (the
+ // latter can happen under schema evolution: a column is in the requested schema but not in this
+ // physical parquet file). For any non-primitive key we still bail; phase-1 reads only primitive
+ // leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> missingKeyLocalPositions = new ArrayList<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals. Fail loudly
+ // rather than dropping the filter -- extractStorageFilters already removed it from the
+ // post-scan Filter, so silently ignoring it here would return wrong rows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create. Fail loudly for
+ // the same reason as above.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (missingColumns.contains(column)) {
+ missingKeyLocalPositions.add(i);
+ } else {
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // If any key column is missing from this file, rewrite the predicate to substitute the constant
+ // the reader will actually materialize for that column. That is the column's existence DEFAULT
+ // when it has one (ParquetColumnVector writes it into the output vector and marks the vector
+ // constant), otherwise null. Substituting null for a column that reads back as its default
+ // would filter on a value the scan never returns. The predicate must be evaluated against the
+ // substituted constant rather than skipped: null does not always mean false in a filter.
+ if (!missingKeyLocalPositions.isEmpty()) {
+ int[] missing = new int[missingKeyLocalPositions.size()];
+ Object[] missingValues = new Object[missingKeyLocalPositions.size()];
+ Object[] existenceDefaults =
+ ResolveDefaultColumns.existenceDefaultValues(sparkRequestedSchema);
+ for (int i = 0; i < missing.length; i++) {
+ missing[i] = missingKeyLocalPositions.get(i);
+ missingValues[i] = existenceDefaults[keyIndices[missing[i]]];
+ }
+ storageFilter = storageFilter.rewriteForMissingKeys(missing, missingValues);
+
+ if (presentKeyColumns.isEmpty()) {
+ // All key columns missing: the rewritten predicate is fully constant. Evaluate once and
+ // apply uniformly to the whole file.
+ boolean keepAll = storageFilter.evalAllMissing();
+ if (keepAll) {
+ // Predicate is constant-true for this file: no filtering to do.
+ storageFilter = null;
+ } else {
+ // Predicate is constant-false/null: no row from this file can pass the filter.
+ storageFilter = null;
+ hitEndOfData = true;
+ }
+ return;
+ }
+ }
+
+ keyDescriptors = new ColumnDescriptor[presentKeyColumns.size()];
+ keyRequired = new boolean[presentKeyColumns.size()];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < presentKeyColumns.size(); i++) {
+ ParquetColumn column = presentKeyColumns.get(i);
+ keyDescriptors[i] = column.descriptor().get();
+ keyRequired[i] = column.required();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyRequestedSchema = keySchemaBuilder.named(requestedSchema.getName());
+
+ // Build the non-key (complement) schema. When the projection has at least one non-key column,
+ // phase 2 will switch lateMatReader's schema to this and read those columns under finalRanges.
+ // When the projection is *all* key columns (e.g. a scan whose only column is the bloom probe),
+ // splicing still pays off: phase 2 has nothing useful to read, so we skip it entirely (no read,
+ // no IO) and emit batches purely from the key queues. The `nonKeyRequestedSchema != null` check
+ // downstream gates phase-2 IO.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount > 0) {
+ nonKeyRequestedSchema = nonKeyBuilder.named(requestedSchema.getName());
+ requireOffsetIndexesForPhase2();
+ }
+ initializeSplicingState(presentKeyColumns);
+
+ totalBlockCount = lateMatReader.getRowGroups().size();
+ nextBlockIndex = 0;
+ }
+
+ /**
+ * Fails now if any projected column of any row group lacks a Parquet offset index.
+ *
+ * <p>Phase 2 reads a strict subset of a row group's rows, which parquet can only do via the
+ * offset index; files written before parquet-mr 1.11, or by writers that omit it, have none. We
+ * cannot
+ * widen phase 2 to the whole block instead, because the key vectors already hold only the
+ * survivors and the batch would misalign -- and we cannot skip the filter either, since
+ * {@code extractStorageFilters} has already removed it from the post-scan Filter.
+ *
+ * <p>Every projected column is checked, not just the non-key ones phase 2 reads, because parquet
+ * builds one column index store per row group and reuses it. Phase 0 asks for the row ranges
+ * under the full requested schema, so {@code ColumnIndexStoreImpl.create} is called with the key
+ * columns in its path set, and it returns its {@code EMPTY} singleton as soon as any one of those
+ * paths has no offset index. {@code ParquetFileReader.getColumnIndexStore} memoizes that store
+ * per block, and {@code EMPTY.getOffsetIndex} throws for *every* column. So a key column with no
+ * offset index kills phase 2 too, with a raw {@code MissingOffsetIndexException} naming some
+ * non-key column and none of the guidance below.
+ *
+ * <p>Checking up front rather than at the first partially-kept row group is deliberate: whether
+ * phase 2 needs the offset index otherwise depends on how selective the filter turns out to be on
+ * this particular file, so the same query would fail or not depending on the data. This is
+ * conservative -- a filter that happens to keep every row of every block would not have needed
+ * the offset index -- but such a filter also saves nothing, so failing loudly loses nothing.
+ *
+ * <p>The check itself is free: {@code getOffsetIndexReference()} is a footer field that
+ * {@link #initialize} has already read.
+ */
+ private void requireOffsetIndexesForPhase2() {
+ Set<ColumnPath> projectedPaths = new HashSet<>();
+ for (ColumnDescriptor column : requestedSchema.getColumns()) {
+ projectedPaths.add(ColumnPath.get(column.getPath()));
+ }
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ for (ColumnChunkMetaData chunk : blocks.get(blockIdx).getColumns()) {
+ if (projectedPaths.contains(chunk.getPath()) && chunk.getOffsetIndexReference() == null) {
Review Comment:
**This turns a performance feature into an availability one.**
Any Parquet file without an offset index fails the query outright, with no per-file fallback. That is not an exotic shape: pyarrow's `write_table` defaults to `write_page_index=False`, Impala omits it, and parquet-mr < 1.11 (Spark <= 2.4) has no column/offset index at all.
Scenario: an operator sets `spark.sql.parquet.storageFilterPushdown.enabled=true` as a cluster default. Any join where `InjectRuntimeFilter` attaches a bloom to such a table now throws from `initialize()` on every task, retries 4x, and fails the stage -- where before this PR the bloom was an ordinary post-scan `FilterExec` and the query worked. One legacy file in one partition is enough.
Two things make it sting more than it needs to: the check is eager over every row group even when the filter keeps 100% of rows and phase 2 would never consult the index, and the only remedy the message offers is a global conf flip.
The reader already has precedent for degrading per file -- L731-741 sets `storageFilter = null` when every key column is missing. The same shape would work here: fall back to the eager read for this file and evaluate the predicate during splice. That keeps the "extraction already removed the conjunct" invariant intact while leaving the feature purely optional.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +929,450 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+
+ // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the
+ // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and
+ // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes
+ // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index
+ // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's
+ // ranges under the full schema, and with it off we do not call getRowRanges at all.
+ lateMatReader.setRequestedSchema(requestedSchema);
+ // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT
+ // options.useColumnIndexFilter(), so calling it unconditionally would keep applying
+ // column-index filtering after a user turned it off. That conf is the documented escape hatch
+ // for files whose column index is wrong, and trusting a wrong column index here would drop
+ // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no
+ // longer holds this predicate.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ if (pushedFilterRanges.rowCount() == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // Both byte metrics are always wired in production (FileSourceScanLike creates all five
+ // whenever storageFilters is non-empty), so this only skips the work on the test-only path
+ // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so
+ // there is nothing here to avoid on the production path.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ long baselineRows = pushedFilterRanges.rowCount();
+ long baselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema,
+ pushedFilterRanges)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyRequestedSchema);
+ long phase1Bytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema,
+ pushedFilterRanges)
+ : 0L;
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite "
+ + pushedFilterRanges.rowCount() + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+
+ if (finalRanges.rowCount() == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely. We still
+ // paid phase-1 to read the key column, so the bytes avoided vs a no-storage-filter read
+ // are baseline - phase1 (the non-key bytes the no-filter path would have read).
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(baselineBytes - phase1Bytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyRequestedSchema is null); emit reconstructs each
+ // batch from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyRequestedSchema == null) {
+ keptRows = finalRanges.rowCount();
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(nonKeyRequestedSchema);
+ // requireOffsetIndexesForPhase2() already established that every projected column of every
+ // row group has an offset index, so this page-filtering read cannot fail for want of one.
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRanges.rowCount()
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ phase2Bytes = bytesAvoidedPf != null
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, nonKeyRequestedSchema,
+ finalRanges)
+ : 0L;
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(baselineBytes - phase1Bytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the leaf columns of {@code schema} when it reads
+ * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are
+ * included, because both are read whenever any page of a chunk is read.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them. Every caller for a given block walks the same
+ * metadata, so a skipped column drops out of the baseline and the per-phase totals alike.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ MessageType schema,
+ RowRanges rowRanges) {
+ if (schema == null || rowRanges.rowCount() == 0 || schema.getColumns().isEmpty()) {
+ return 0L;
+ }
+ BlockMetaData block = reader.getRowGroups().get(blockIndex);
+ long blockRowCount = block.getRowCount();
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : block.getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ boolean wholeBlock = rowRanges.rowCount() == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : schema.getColumns()) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Reads all rows of the given key-only {@link PageReadStore} (which contains only rows in
+ * {@code pushedFilterRanges}) in capacity-sized chunks, evaluates the storage filter on each row,
+ * builds a {@link RowRanges} of surviving rows in original block-row coordinates, and appends
+ * survivor key values into the per-key-column accumulators ({@link #currentKeyAccumulators}).
+ * When an accumulator hits {@link #capacity}, it's pushed into {@link #keyVectorQueues} and a
+ * fresh one is allocated. After all rows have been examined, any partial trailing accumulator is
+ * pushed too.
+ *
+ * <p>The result is a subset of {@code pushedFilterRanges}: rows not in {@code
+ * pushedFilterRanges} were never read and are implicitly excluded.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ long keyRowsTotal = pushedFilterRanges.rowCount();
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ long remaining = keyRowsTotal;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ appendSurvivorRowToAccumulators(r);
+ }
+ }
+ remaining -= num;
+ }
+
+ finalizePartialAccumulators();
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
Review Comment:
**If this allocation loop throws, `close()` cannot reach what it already allocated.**
`keyScratchBatch` is assigned only after the loop completes, and `close()` (L338-342) frees the scratch vectors solely through `keyScratchBatch`. So an off-heap allocation failure on the second iteration leaves `keyScratchVectors` non-null holding a live `OffHeapColumnVector` while `keyScratchBatch` is still null -- `close()` tests `keyScratchBatch != null`, finds null, and skips them.
Secondarily, the `if (keyScratchVectors != null) return;` guard means a later call would proceed with a partially populated array and NPE at `readers[i].readBatch(num, keyScratchVectors[i], ...)`.
Closing `keyScratchVectors` element-wise, the way `closeSplicingState()` already does for the accumulators, removes both.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -421,11 +552,313 @@ public boolean nextBatch() throws IOException {
return true;
}
+ /**
+ * Splicing emit path. Closes the previous emit's dequeued key vectors (releasing survivor memory
+ * incrementally), advances to the next row group if needed via {@link #checkEndOfRowGroup()},
+ * dequeues one survivor key vector per key column, drives non-key column readers for {@code num}
+ * rows, and assembles a fresh {@link ColumnarBatch} interleaving key (dequeued) and non-key
+ * (persistent value-vector) slots in the original projection order.
+ * The per-emit batch is a transient view over vectors owned elsewhere; see {@link #close()} and
+ * {@link #closeSplicingState()}.
+ */
+ private boolean nextBatchSplicing() throws IOException {
+ if (pendingCloseKeyVectors != null) {
+ for (WritableColumnVector v : pendingCloseKeyVectors) {
+ if (v != null) v.close();
+ }
+ pendingCloseKeyVectors = null;
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) continue;
+ columnVectors[i].reset();
+ }
+ // Match the eager path and zero the outgoing batch before the terminal checks below. Without
+ // this, a terminal call leaves `columnarBatch` pointing at the previous emit whose key slots
+ // were just closed above -- with off-heap vectors those buffers are already freed, so a
+ // consumer that read the batch after nextBatch() returned false would see freed memory.
+ if (columnarBatch != null) columnarBatch.setNumRows(0);
+ if (hitEndOfData) return false;
+ if (rowsReturned >= totalRowCount) return false;
+ checkEndOfRowGroup();
+ if (hitEndOfData) return false;
+
+ int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned);
+
+ WritableColumnVector[] dequeued = new WritableColumnVector[keyVectorQueues.length];
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ dequeued[i] = keyVectorQueues[i].removeFirst();
+ }
+
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) continue;
+ ParquetColumnVector cv = columnVectors[i];
+ for (ParquetColumnVector leafCv : cv.getLeaves()) {
+ VectorizedColumnReader columnReader = leafCv.getColumnReader();
+ if (columnReader != null) {
+ columnReader.readBatch(num, leafCv.getValueVector(),
+ leafCv.getRepetitionLevelVector(), leafCv.getDefinitionLevelVector());
+ }
+ }
+ cv.assemble();
+ }
+ if (rowIndexGenerator != null) {
+ // Row-index column is identified by name (ROW_INDEX_TEMPORARY_COLUMN_NAME), which is a
+ // synthetic metadata column never referenced by a storage filter, so its slot is a non-key
+ // slot with a persistent ParquetColumnVector.
+ rowIndexGenerator.populateRowIndex(columnVectors, num);
+ }
+
+ ColumnVector[] cols = new ColumnVector[persistentBatchColumns.length];
+ // This walks batch slots in ascending order while `keyIdx` walks the survivor queues in
+ // key-row-position order, so it pairs the k-th smallest key slot with key-row position k.
+ // That is only the identity because `ParquetStorageFilter.create` sorts `keyColumnIndices`
+ // ascending -- see the comment there. `isKeyTopLevel` marks which slots are keys but not their
+ // position in that list, so this loop cannot reconstruct the pairing on its own: if the list
+ // ever stops being sorted, key columns silently swap places in the output batch.
+ int keyIdx = 0;
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ if (i < isKeyTopLevel.length && isKeyTopLevel[i]) {
+ cols[i] = dequeued[keyIdx++];
+ } else {
+ cols[i] = persistentBatchColumns[i];
+ }
+ }
+ columnarBatch = new ColumnarBatch(cols);
Review Comment:
**This breaks `resultBatch()`'s documented contract.**
`resultBatch()`'s javadoc says "Returns the ColumnarBatch object that will be used for all rows returned by this reader. **This object is reused.**" Under splicing a new `ColumnarBatch` is created per batch, so a caller that follows the documented hoist-once pattern reads zero rows forever -- no exception, no wrong values, just silently empty.
That pattern exists in-tree: `DataSourceReadBenchmark.scala:234` and `ParquetEncodingSuite.scala:74` both do `val batch = reader.resultBatch()` outside the `while (reader.nextBatch())` loop. Neither uses a storage filter today, and production v1 paths are safe because `RecordReaderIterator.next()` re-reads `getCurrentValue` per batch -- but `resultBatch()` is public and this suite's own helpers deliberately re-fetch it inside the loop, which suggests the constraint was already noticed.
Either update the javadoc, or keep one stable `ColumnarBatch` and mutate its column array in place.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -151,6 +152,63 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
}
}
+ /**
+ * Splits `afterScanFilters` into bloom-filter conjuncts that can be pushed to the storage layer
+ * for late materialization (returned as the first element) and the remaining filters that stay as
+ * a post-scan FilterExec (returned as the second element).
+ *
+ * Eligibility (statically checked here so the runtime never silently loses the filter):
+ * - The storage-filter pushdown SQL conf is on.
+ * - The file format is exactly [[ParquetFileFormat]]. Subclasses are excluded on purpose: they
+ * may customize reading by overriding `buildReaderWithPartitionValues`, and attaching storage
+ * filters would route the scan through `ParquetFileFormat`'s own reader instead, silently
+ * dropping whatever the subclass does.
+ * - The vectorized reader is feasible for the schema the reader will actually see, i.e.
+ * `partitionSchema ++ outputDataSchema` -- the same schema `ParquetFileFormat.buildReader`
+ * derives `enableVectorizedReader` from.
+ * - The conjunct is a top-level [[BloomFilterMightContain]] (not nested under OR/NOT).
+ * - The conjunct is deterministic. `ParquetStorageFilter.test` evaluates the predicate without
+ * calling `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a
+ * `Nondeterministic` expression, so a non-deterministic conjunct would fail at task time. No
+ * such bloom exists today, because the only producer is `InjectRuntimeFilter` and a join key
+ * is deterministic, but this gate should not depend on a distant rule.
+ * - The bloom's value-side references are projected data columns whose type the reader's value
+ * copier supports (see [[ParquetStorageFilter.isSupportedKeyType]]).
+ *
+ * If any condition fails, ALL bloom filters stay in the second element to preserve the existing
+ * fallback behavior.
+ */
+ private def extractStorageFilters(
+ afterScanFilters: ExpressionSet,
+ fsRelation: HadoopFsRelation,
+ readDataColumns: Seq[Attribute],
+ outputDataSchema: StructType): (Seq[Expression], ExpressionSet) = {
+ val sparkSession = fsRelation.sparkSession
+ val sqlConf = sparkSession.sessionState.conf
+ if (!sqlConf.parquetStorageFilterPushdownEnabled) return (Nil, afterScanFilters)
+ if (fsRelation.fileFormat.getClass != classOf[ParquetFileFormat]) {
Review Comment:
**Could the format answer this instead of the planner type-checking it?**
This makes the generic `FileSourceStrategy` import `ParquetFileFormat` and `ParquetStorageFilter` (L36) and hard-code an exact-class check, so ORC or any third-party format can only participate by editing the planner, and `ParquetFileFormat` subclasses have no opt-in at all.
L205 goes a step further: `ParquetStorageFilter.isSupportedKeyType` is the set of types `VectorizedParquetRecordReader.copierFor` can copy -- a reader implementation detail the planner now owns, with a "must stay in lockstep" invariant spanning two packages and enforced only by comment.
Spark's established shape for this is a capability method on `FileFormat`, exactly like `supportBatch`, `supportDataType`, `vectorTypes` and `metadataSchemaFields` -- and `supportBatch` is already being called polymorphically five lines below. Something like:
```scala
def supportsStorageFilter(expr: Expression, readDataSchema: StructType): Boolean = false
```
would absorb both checks and keep the type list inside the parquet package.
The justification given above is that subclasses may override `buildReaderWithPartitionValues` -- but that concern is a consequence of this PR inverting the delegation, and it can be expressed inside the override, in the class that knows why: `getClass == classOf[ParquetFileFormat] && ...`.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +929,450 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+
+ // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the
+ // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and
+ // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes
+ // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index
+ // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's
+ // ranges under the full schema, and with it off we do not call getRowRanges at all.
+ lateMatReader.setRequestedSchema(requestedSchema);
+ // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT
+ // options.useColumnIndexFilter(), so calling it unconditionally would keep applying
+ // column-index filtering after a user turned it off. That conf is the documented escape hatch
+ // for files whose column index is wrong, and trusting a wrong column index here would drop
+ // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no
+ // longer holds this predicate.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ if (pushedFilterRanges.rowCount() == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // Both byte metrics are always wired in production (FileSourceScanLike creates all five
+ // whenever storageFilters is non-empty), so this only skips the work on the test-only path
+ // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so
+ // there is nothing here to avoid on the production path.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ long baselineRows = pushedFilterRanges.rowCount();
+ long baselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema,
+ pushedFilterRanges)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyRequestedSchema);
+ long phase1Bytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema,
+ pushedFilterRanges)
+ : 0L;
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite "
+ + pushedFilterRanges.rowCount() + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+
+ if (finalRanges.rowCount() == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely. We still
+ // paid phase-1 to read the key column, so the bytes avoided vs a no-storage-filter read
+ // are baseline - phase1 (the non-key bytes the no-filter path would have read).
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(baselineBytes - phase1Bytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyRequestedSchema is null); emit reconstructs each
+ // batch from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyRequestedSchema == null) {
+ keptRows = finalRanges.rowCount();
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(nonKeyRequestedSchema);
+ // requireOffsetIndexesForPhase2() already established that every projected column of every
+ // row group has an offset index, so this page-filtering read cannot fail for want of one.
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRanges.rowCount()
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ phase2Bytes = bytesAvoidedPf != null
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, nonKeyRequestedSchema,
+ finalRanges)
+ : 0L;
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(baselineBytes - phase1Bytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the leaf columns of {@code schema} when it reads
+ * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are
+ * included, because both are read whenever any page of a chunk is read.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them. Every caller for a given block walks the same
+ * metadata, so a skipped column drops out of the baseline and the per-phase totals alike.
+ */
+ private static long compressedBytesForRowRanges(
Review Comment:
**`MessageType.getColumns()` is rebuilt ~9x per row group, for two metrics.**
`getColumns()` is not cached in parquet-mr (its source carries a `// TODO: optimize this`): each call rebuilds `getPaths(0)` and, per leaf, does `getType(path)` + `getMaxRepetitionLevel` + `getMaxDefinitionLevel` tree descents plus a `new ColumnDescriptor`.
`ParquetFileReader.setRequestedSchema(MessageType)` is `setRequestedSchema(projection.getColumns())`, so L961/L993/L1031 contribute three rebuilds per row group. This method calls `schema.getColumns()` twice per invocation (L1099, L1111) across three invocations (L986, L995, L1043) for six more. It also rebuilds a `HashMap<ColumnPath, ColumnChunkMetaData>` over *every* chunk of the block rather than the projected ones (L1104-1107), and `rowRanges.rowCount()` -- O(#ranges), and `finalRanges` can hold ~250K ranges for a 1M-row block at 50% selectivity -- is recomputed at L1009, L1099 and L1108.
For a 200-column projection on a 2000-column file that is roughly 2,400 allocations, 4,800 tree descents and 6,000 `HashMap.put`s per row group, to populate two counters.
Hoisting the three `List<ColumnDescriptor>` into fields in `initializeLateMaterialization()` (and using the `setRequestedSchema(List<ColumnDescriptor>)` overload), building the path->chunk map once per block, and passing the already-computed row counts in removes all of it with no behavior change.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +929,450 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+
+ // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the
+ // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and
+ // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes
+ // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index
+ // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's
+ // ranges under the full schema, and with it off we do not call getRowRanges at all.
+ lateMatReader.setRequestedSchema(requestedSchema);
+ // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT
+ // options.useColumnIndexFilter(), so calling it unconditionally would keep applying
+ // column-index filtering after a user turned it off. That conf is the documented escape hatch
+ // for files whose column index is wrong, and trusting a wrong column index here would drop
+ // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no
+ // longer holds this predicate.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ if (pushedFilterRanges.rowCount() == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // Both byte metrics are always wired in production (FileSourceScanLike creates all five
+ // whenever storageFilters is non-empty), so this only skips the work on the test-only path
+ // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so
+ // there is nothing here to avoid on the production path.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ long baselineRows = pushedFilterRanges.rowCount();
+ long baselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema,
+ pushedFilterRanges)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyRequestedSchema);
+ long phase1Bytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema,
+ pushedFilterRanges)
+ : 0L;
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite "
+ + pushedFilterRanges.rowCount() + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+
+ if (finalRanges.rowCount() == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely. We still
+ // paid phase-1 to read the key column, so the bytes avoided vs a no-storage-filter read
+ // are baseline - phase1 (the non-key bytes the no-filter path would have read).
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(baselineBytes - phase1Bytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyRequestedSchema is null); emit reconstructs each
+ // batch from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyRequestedSchema == null) {
+ keptRows = finalRanges.rowCount();
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(nonKeyRequestedSchema);
+ // requireOffsetIndexesForPhase2() already established that every projected column of every
+ // row group has an offset index, so this page-filtering read cannot fail for want of one.
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRanges.rowCount()
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ phase2Bytes = bytesAvoidedPf != null
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, nonKeyRequestedSchema,
+ finalRanges)
+ : 0L;
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(baselineBytes - phase1Bytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the leaf columns of {@code schema} when it reads
+ * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are
+ * included, because both are read whenever any page of a chunk is read.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them. Every caller for a given block walks the same
+ * metadata, so a skipped column drops out of the baseline and the per-phase totals alike.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ MessageType schema,
+ RowRanges rowRanges) {
+ if (schema == null || rowRanges.rowCount() == 0 || schema.getColumns().isEmpty()) {
+ return 0L;
+ }
+ BlockMetaData block = reader.getRowGroups().get(blockIndex);
+ long blockRowCount = block.getRowCount();
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : block.getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ boolean wholeBlock = rowRanges.rowCount() == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : schema.getColumns()) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Reads all rows of the given key-only {@link PageReadStore} (which contains only rows in
+ * {@code pushedFilterRanges}) in capacity-sized chunks, evaluates the storage filter on each row,
+ * builds a {@link RowRanges} of surviving rows in original block-row coordinates, and appends
+ * survivor key values into the per-key-column accumulators ({@link #currentKeyAccumulators}).
+ * When an accumulator hits {@link #capacity}, it's pushed into {@link #keyVectorQueues} and a
+ * fresh one is allocated. After all rows have been examined, any partial trailing accumulator is
+ * pushed too.
+ *
+ * <p>The result is a subset of {@code pushedFilterRanges}: rows not in {@code
+ * pushedFilterRanges} were never read and are implicitly excluded.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ long keyRowsTotal = pushedFilterRanges.rowCount();
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ long remaining = keyRowsTotal;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ appendSurvivorRowToAccumulators(r);
+ }
+ }
+ remaining -= num;
+ }
+
+ finalizePartialAccumulators();
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates the per-key-column accumulator vectors if any slot is null (i.e. the previous
+ * accumulator was just pushed to the queue or this is the first row group). Each accumulator has
+ * {@link #capacity} rows.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of each {@link #keyScratchVectors} into the corresponding
+ * {@link #currentKeyAccumulators}. When the accumulators fill, they're pushed onto their queues
+ * and fresh ones allocated. All key columns are appended in lockstep so accumulators stay
+ * aligned.
+ */
+ private void appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ }
+ }
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+ }
+ }
+
+ /**
+ * Pushes any partially-filled accumulator into its queue at row-group end so the emit path can
+ * dequeue it as the row group's final batch.
+ */
+ private void finalizePartialAccumulators() {
+ if (currentKeyAccumulatorRowCount == 0) return;
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Closes anything held by the splicing path: pending dequeued key vectors not yet rolled over,
+ * any vectors still queued (e.g. on early termination), and partially-filled accumulators.
+ * Called from {@link #close()}.
+ */
+ private void closeSplicingState() {
+ if (pendingCloseKeyVectors != null) {
+ for (WritableColumnVector v : pendingCloseKeyVectors) {
+ if (v != null) v.close();
+ }
+ pendingCloseKeyVectors = null;
+ }
+ if (keyVectorQueues != null) {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ if (q != null) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ }
+ }
+ if (currentKeyAccumulators != null) {
+ for (WritableColumnVector v : currentKeyAccumulators) {
+ if (v != null) v.close();
+ }
+ currentKeyAccumulators = null;
+ }
+ }
+
+ /**
+ * Per-key-column value copier: appends one value from {@code src[srcRow]} to
+ * {@code dst[dstRow]}. Picked once at init via {@link #copierFor(DataType)}; called per surviving
+ * row in {@link #appendSurvivorRowToAccumulators}. Caller handles null sources.
+ */
+ @FunctionalInterface
+ private interface ValueCopier {
+ void copy(WritableColumnVector dst, int dstRow, WritableColumnVector src, int srcRow);
+ }
+
+ /**
+ * Returns a {@link ValueCopier} for the given key {@link DataType}. The set of types handled here
+ * is the definition behind {@code ParquetStorageFilter.isSupportedKeyType}, which gates both
+ * planning-time extraction and {@code ParquetStorageFilter.create} -- so the throw at the end is
+ * unreachable. Teach both sides at once when adding a type; a type admitted there but missing
+ * here becomes a task failure instead of a planning-time rejection.
+ */
+ private static ValueCopier copierFor(DataType dt) {
Review Comment:
**This duplicates `RowToColumnConverter`, and has already drifted.**
`RowToColumnConverter.getConverterForType` (`sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala:288-327`) has the identical groupings -- `IntegerType | DateType | _: YearMonthIntervalType` -> Int, `LongType | TimestampType | TimestampNTZType | _: DayTimeIntervalType | _: TimeType` -> Long, and the same `MAX_INT_DIGITS` / `MAX_LONG_DIGITS` three-way decimal split. `BasicNullableTypeConverter` also already handles the `isNullAt` -> `putNull` branch at L1249.
The hot loop already has an `InternalRow` in hand (`keyScratchBatch.getRow(r)`), so `convert(row, currentKeyAccumulators)` is close to a drop-in, and `ColumnVectorUtils.java:134-138` is the existing precedent for calling this `private[execution]` Scala class from Java in this module.
The drift is visible in the next block: the `VarcharType` / `CharType` branches at L1367-1368 are dead, since both extend `StringType` (`sql/api/.../CharType.scala:34`, `VarcharType.scala:33`) and the `StringType` test comes first in the same `||` chain. `ParquetStorageFilter.isSupportedKeyType`'s comment at L216 states this correctly, so the two files already disagree about the same fact -- which is the "keep them in lockstep" cost the javadoc above warns about.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java:
##########
@@ -87,6 +87,13 @@ public abstract class SpecificParquetRecordReaderBase<T> extends RecordReader<Vo
protected ParquetRowGroupReader reader;
+ /**
+ * The opened input file and parquet footer. Stored so subclasses can read footer-derived metadata
+ * without re-opening the file. Set by both {@link #initialize} overloads.
+ */
+ protected HadoopInputFile inputFile;
Review Comment:
**These two fields are never read, and the javadoc is inaccurate.**
Repo-wide: `inputFile` is assigned at L120, L125 and L193 and read nowhere; `fileFooter`'s only read is `fileFooter.getFileMetaData().getSchema()` at L197, inside the same method that assigns it, where the pre-existing `fileReader.getFooter()` expression was equivalent. The late-materialization path gets what it needs from `reader.getUnderlyingReader()` / `lateMatReader.getFile()` instead.
The doc also says "Set by both `initialize` overloads" -- there are four, and the `@VisibleForTesting initialize(MessageType, MessageType, ParquetRowGroupReader, int)` at L226 (the one `ParquetVectorizedSuite.scala:802` uses) leaves both null, so a subclass trusting the doc would NPE.
One further side effect: `close()` nulls only `reader`, so `fileFooter` now pins the whole `ParquetMetadata` -- every `BlockMetaData` and column-chunk statistic -- for the reader's lifetime on *every* Parquet scan, not just storage-filter ones.
Dropping the hunk looks right; `fileFooter` can stay a local.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +929,450 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+
+ // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the
+ // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and
+ // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes
+ // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index
+ // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's
+ // ranges under the full schema, and with it off we do not call getRowRanges at all.
+ lateMatReader.setRequestedSchema(requestedSchema);
+ // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT
+ // options.useColumnIndexFilter(), so calling it unconditionally would keep applying
+ // column-index filtering after a user turned it off. That conf is the documented escape hatch
+ // for files whose column index is wrong, and trusting a wrong column index here would drop
+ // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no
+ // longer holds this predicate.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ if (pushedFilterRanges.rowCount() == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // Both byte metrics are always wired in production (FileSourceScanLike creates all five
+ // whenever storageFilters is non-empty), so this only skips the work on the test-only path
+ // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so
+ // there is nothing here to avoid on the production path.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ long baselineRows = pushedFilterRanges.rowCount();
+ long baselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema,
+ pushedFilterRanges)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyRequestedSchema);
+ long phase1Bytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema,
+ pushedFilterRanges)
+ : 0L;
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite "
+ + pushedFilterRanges.rowCount() + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+
+ if (finalRanges.rowCount() == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely. We still
+ // paid phase-1 to read the key column, so the bytes avoided vs a no-storage-filter read
+ // are baseline - phase1 (the non-key bytes the no-filter path would have read).
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(baselineBytes - phase1Bytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyRequestedSchema is null); emit reconstructs each
+ // batch from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyRequestedSchema == null) {
+ keptRows = finalRanges.rowCount();
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(nonKeyRequestedSchema);
+ // requireOffsetIndexesForPhase2() already established that every projected column of every
+ // row group has an offset index, so this page-filtering read cannot fail for want of one.
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRanges.rowCount()
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ phase2Bytes = bytesAvoidedPf != null
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, nonKeyRequestedSchema,
+ finalRanges)
+ : 0L;
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(baselineBytes - phase1Bytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the leaf columns of {@code schema} when it reads
+ * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are
+ * included, because both are read whenever any page of a chunk is read.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them. Every caller for a given block walks the same
+ * metadata, so a skipped column drops out of the baseline and the per-phase totals alike.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ MessageType schema,
+ RowRanges rowRanges) {
+ if (schema == null || rowRanges.rowCount() == 0 || schema.getColumns().isEmpty()) {
+ return 0L;
+ }
+ BlockMetaData block = reader.getRowGroups().get(blockIndex);
+ long blockRowCount = block.getRowCount();
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : block.getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ boolean wholeBlock = rowRanges.rowCount() == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : schema.getColumns()) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Reads all rows of the given key-only {@link PageReadStore} (which contains only rows in
+ * {@code pushedFilterRanges}) in capacity-sized chunks, evaluates the storage filter on each row,
+ * builds a {@link RowRanges} of surviving rows in original block-row coordinates, and appends
+ * survivor key values into the per-key-column accumulators ({@link #currentKeyAccumulators}).
+ * When an accumulator hits {@link #capacity}, it's pushed into {@link #keyVectorQueues} and a
+ * fresh one is allocated. After all rows have been examined, any partial trailing accumulator is
+ * pushed too.
+ *
+ * <p>The result is a subset of {@code pushedFilterRanges}: rows not in {@code
+ * pushedFilterRanges} were never read and are implicitly excluded.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ long keyRowsTotal = pushedFilterRanges.rowCount();
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
Review Comment:
**Unbounded, and invisible to Spark's memory manager.**
This loop evaluates the entire row group before `loadNextRowGroupWithLateMaterialization` returns, so `keyVectorQueues` holds every surviving key value of one row group at once -- and those vectors are live at the same time as phase 2's non-key vectors.
At the default 128MB `parquet.block.size` a row group is commonly 1M+ rows, and blooms are deliberately false-positive-prone, so most survive. For a 32-byte string key at `capacity = 4096` that is ~250 full-width vectors plus their variable-length child buffers, per key column, per concurrent task. With off-heap enabled these go through `Platform.allocateMemory` rather than a `MemoryConsumer`, so the executor gets OOM-killed by YARN/k8s instead of raising `SparkOutOfMemoryError` or spilling.
The conf doc's "a task holds up to one extra copy of the key columns for one row group" is accurate but reads as bounded. It scales with row-group size, not with `capacity`, and for a dictionary-encoded key the accumulators store *expanded* bytes -- a chunk holding 1,000 distinct 40-byte values plus 1M dictionary IDs becomes ~20MB of materialized strings.
Evaluating one `capacity`-sized chunk at a time (emit, then continue) would bound this and fix the `LIMIT` stall below; a guard that falls back to the eager path above some `blockRowCount` would be the cheaper stopgap.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,1965 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayOutputStream, File}
+import java.net.URI
+import java.time.LocalTime
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapreduce.Job
+import org.apache.parquet.hadoop.{ParquetInputFormat, ParquetOutputFormat}
+
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, BloomFilterMightContain, BoundReference, Coalesce, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, XxHash64}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FileSourceScanLike, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests the late-materialization path of [[VectorizedParquetRecordReader]] driven by a
+ * [[ParquetStorageFilter]]. Writes small multi-row-group parquet files, wires a hand-built filter
+ * into the reader, and asserts correctness + the two storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession {
+ import testImplicits._
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = rows.toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. Note this does NOT disable the
+ // column index, despite what an earlier version of this comment claimed.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Collects all rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) = {
+ val reader = new VectorizedParquetRecordReader(false, 4096)
+ reader.setStorageFilter(storageFilter)
+ reader.initialize(filePath, java.util.Arrays.asList("k", "v"))
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch().asInstanceOf[ColumnarBatch]
+ val n = batch.numRows()
+ val kVec = batch.column(0)
+ val vVec = batch.column(1)
+ var i = 0
+ while (i < n) {
+ collected += ((kVec.getLong(i), vVec.getUTF8String(i).toString))
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
Review Comment:
**The drain happens outside the callers' `try`/`finally`, so the reader leaks on exactly the failures these tests exist to catch.**
This helper (and `readKeyOnlyAll` L152, `readAllWith` L771, `survivorsViaPlainPath` L805) runs the whole `while (reader.nextBatch())` loop and only then returns the reader. Callers wrap just the assertions in `try { ... } finally { reader.close() }`, so the `finally` protects nothing that can realistically fail.
Concretely: the off-heap parameterized test at L1021-1041 calls `readAllWith(..., useOffHeap = true)`. A misaligned-queue regression would surface as `NoSuchElementException` from `keyVectorQueues[i].removeFirst()` inside `nextBatch()` -- and then the reader is never closed, leaking the `ParquetFileReader`, its `SeekableInputStream`, the off-heap `persistentBatchColumns`, the survivor queues and the scratch vectors for the rest of the JVM, which can cascade into unrelated failures in the same suite. `survivorsViaPlainPath` is the same shape and is the oracle for ~35 parameterized key-type tests, and its `extract` can `fail(...)` from inside the loop.
Wrapping the drain in `Utils.tryWithSafeFinally`, or having the helper own `close()` and return only the rows, fixes all four.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala:
##########
@@ -165,6 +166,47 @@ trait FileFormat {
}
}
+ /**
+ * Like [[buildReaderWithPartitionValues]] but additionally accepts a sequence of storage filters:
+ * Catalyst expressions that the storage layer may evaluate to drive value-column IO pruning based
+ * on key-column evaluation (e.g., late materialization with a runtime bloom filter).
+ *
+ * The default implementation delegates to [[buildReaderWithPartitionValues]] and accepts no
+ * storage filter at all. File formats that support storage-filter pushdown (e.g., Parquet) should
+ * override this method.
+ *
+ * A non-empty `storageFilters` here is a planner bug, so the default body rejects it rather than
+ * dropping it. The planner removes an extracted conjunct from the post-scan `Filter`, so a reader
+ * that ignores it returns rows the filter rejects. Every other layer of the feature fails loudly
+ * for the same reason. Unreachable today, because `FileSourceStrategy.extractStorageFilters` only
+ * extracts for `ParquetFileFormat` itself, but that keeps the invariant one method away from the
+ * code that relies on it.
+ *
+ * Scalar subqueries inside `storageFilters` are expected to have been materialized before this
+ * method is called, so that the returned reader can be safely serialized to executors.
+ *
+ * `storageFilterMetrics` is an optional map of SQL metrics the reader can update during execution
+ * (e.g. number of row groups skipped). The scan is expected to expose these metrics via its
+ * `metrics` field so they show up in the SQL UI.
+ */
+ def buildReaderWithStorageFilters(
+ sparkSession: SparkSession,
+ dataSchema: StructType,
+ partitionSchema: StructType,
+ requiredSchema: StructType,
+ filters: Seq[Filter],
+ storageFilters: Seq[Expression],
+ options: Map[String, String],
+ hadoopConf: Configuration,
+ storageFilterMetrics: Map[String, SQLMetric] = Map.empty
+ ): PartitionedFile => Iterator[InternalRow] = {
+ require(storageFilters.isEmpty,
+ s"${getClass.getSimpleName} does not support storage-filter pushdown, but was given " +
+ storageFilters.mkString("[", ", ", "]"))
+ buildReaderWithPartitionValues(
Review Comment:
**Mutual recursion with `ParquetFileFormat`.**
This default calls `buildReaderWithPartitionValues`, and `ParquetFileFormat.buildReaderWithPartitionValues` (L195) now delegates to `buildReaderWithStorageFilters`. That call is unqualified and neither method is `final`, so it dispatches virtually.
A subclass of `ParquetFileFormat` that opts out of storage filters with the most natural body -- the same one this default uses:
```scala
override def buildReaderWithStorageFilters(...) =
buildReaderWithPartitionValues(sparkSession, dataSchema, partitionSchema,
requiredSchema, filters, options, hadoopConf)
```
gets `buildReaderWithPartitionValues` -> subclass override -> `buildReaderWithPartitionValues` -> ... and a `StackOverflowError` on the driver the first time `FileSourceScanExec.inputRDD` builds the reader. Even `super.buildReaderWithPartitionValues(...)` recurses, because the super body's unqualified call still lands on the subclass override -- only `super.buildReaderWithStorageFilters(...)` is safe, and the scaladoc does not say so while actively inviting the override ("File formats that support storage-filter pushdown ... should override this method").
Not reachable in-tree, but subclassing `ParquetFileFormat` is a common downstream pattern. Routing `ParquetFileFormat`'s base implementation through a private method, or marking one of the two `final`, closes it cheaply.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -421,11 +552,313 @@ public boolean nextBatch() throws IOException {
return true;
}
+ /**
+ * Splicing emit path. Closes the previous emit's dequeued key vectors (releasing survivor memory
+ * incrementally), advances to the next row group if needed via {@link #checkEndOfRowGroup()},
+ * dequeues one survivor key vector per key column, drives non-key column readers for {@code num}
+ * rows, and assembles a fresh {@link ColumnarBatch} interleaving key (dequeued) and non-key
+ * (persistent value-vector) slots in the original projection order.
+ * The per-emit batch is a transient view over vectors owned elsewhere; see {@link #close()} and
+ * {@link #closeSplicingState()}.
+ */
+ private boolean nextBatchSplicing() throws IOException {
Review Comment:
**Design question: why buffer the survivors instead of re-reading the key columns in phase 2?**
This method, and most of the machinery it depends on, exists only because phase 2 deliberately omits the key columns. If phase 2 set `requestedSchema` instead of `nonKeyRequestedSchema` and read under `finalRanges`, the key columns would come back from the same `PageReadStore`, already aligned with the non-key ones, and the existing eager `nextBatch()` would serve every row group unchanged.
That would delete `nextBatchSplicing` (71 lines), `ValueCopier` + `copierFor` (64), `initializeSplicingState` / `ensureKeyScratchAllocated` / `ensureCurrentKeyAccumulatorsAllocated` / `appendSurvivorRowToAccumulators` / `finalizePartialAccumulators` / `closeSplicingState` (~103), the double-close handling in `close()`, the `skipDataSlots` parameter on `allocateColumns`, eight fields -- and `ParquetStorageFilter.isSupportedKeyType` with the type gate it forces into `FileSourceStrategy`.
It also removes, rather than fixes, several things flagged separately in this review: the whole-row-group buffering, the `LIMIT` stall, the per-row `byte[]` copy, and the "`keyColumnIndices` must stay sorted" coupling that the comment below warns can silently swap key columns in the output batch.
The honest cost is re-reading one narrow column's pages under a subset of the ranges phase 1 already read: always cheaper than phase 1 itself, usually still in page cache, worst case 2x key decode when the filter passes everything. That trades IO for the memory cost the design note at L218-224 already concedes, which seems like the easier direction to defend for a first cut.
If there is a reason this does not work, could it go in the design note? It is the first thing a reader of this method will ask.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +929,450 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+
+ // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the
+ // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and
+ // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes
+ // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index
+ // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's
+ // ranges under the full schema, and with it off we do not call getRowRanges at all.
+ lateMatReader.setRequestedSchema(requestedSchema);
+ // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT
+ // options.useColumnIndexFilter(), so calling it unconditionally would keep applying
+ // column-index filtering after a user turned it off. That conf is the documented escape hatch
+ // for files whose column index is wrong, and trusting a wrong column index here would drop
+ // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no
+ // longer holds this predicate.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ if (pushedFilterRanges.rowCount() == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // Both byte metrics are always wired in production (FileSourceScanLike creates all five
+ // whenever storageFilters is non-empty), so this only skips the work on the test-only path
+ // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so
+ // there is nothing here to avoid on the production path.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ long baselineRows = pushedFilterRanges.rowCount();
+ long baselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema,
+ pushedFilterRanges)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyRequestedSchema);
+ long phase1Bytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema,
+ pushedFilterRanges)
+ : 0L;
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite "
+ + pushedFilterRanges.rowCount() + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+
+ if (finalRanges.rowCount() == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely. We still
+ // paid phase-1 to read the key column, so the bytes avoided vs a no-storage-filter read
+ // are baseline - phase1 (the non-key bytes the no-filter path would have read).
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(baselineBytes - phase1Bytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyRequestedSchema is null); emit reconstructs each
+ // batch from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyRequestedSchema == null) {
+ keptRows = finalRanges.rowCount();
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(nonKeyRequestedSchema);
+ // requireOffsetIndexesForPhase2() already established that every projected column of every
+ // row group has an offset index, so this page-filtering read cannot fail for want of one.
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRanges.rowCount()
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ phase2Bytes = bytesAvoidedPf != null
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, nonKeyRequestedSchema,
+ finalRanges)
+ : 0L;
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(baselineBytes - phase1Bytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the leaf columns of {@code schema} when it reads
+ * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are
+ * included, because both are read whenever any page of a chunk is read.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them. Every caller for a given block walks the same
+ * metadata, so a skipped column drops out of the baseline and the per-phase totals alike.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ MessageType schema,
+ RowRanges rowRanges) {
+ if (schema == null || rowRanges.rowCount() == 0 || schema.getColumns().isEmpty()) {
+ return 0L;
+ }
+ BlockMetaData block = reader.getRowGroups().get(blockIndex);
+ long blockRowCount = block.getRowCount();
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : block.getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ boolean wholeBlock = rowRanges.rowCount() == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : schema.getColumns()) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Reads all rows of the given key-only {@link PageReadStore} (which contains only rows in
+ * {@code pushedFilterRanges}) in capacity-sized chunks, evaluates the storage filter on each row,
+ * builds a {@link RowRanges} of surviving rows in original block-row coordinates, and appends
+ * survivor key values into the per-key-column accumulators ({@link #currentKeyAccumulators}).
+ * When an accumulator hits {@link #capacity}, it's pushed into {@link #keyVectorQueues} and a
+ * fresh one is allocated. After all rows have been examined, any partial trailing accumulator is
+ * pushed too.
+ *
+ * <p>The result is a subset of {@code pushedFilterRanges}: rows not in {@code
+ * pushedFilterRanges} were never read and are implicitly excluded.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ long keyRowsTotal = pushedFilterRanges.rowCount();
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ long remaining = keyRowsTotal;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ appendSurvivorRowToAccumulators(r);
+ }
+ }
+ remaining -= num;
+ }
+
+ finalizePartialAccumulators();
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates the per-key-column accumulator vectors if any slot is null (i.e. the previous
+ * accumulator was just pushed to the queue or this is the first row group). Each accumulator has
+ * {@link #capacity} rows.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of each {@link #keyScratchVectors} into the corresponding
+ * {@link #currentKeyAccumulators}. When the accumulators fill, they're pushed onto their queues
+ * and fresh ones allocated. All key columns are appended in lockstep so accumulators stay
+ * aligned.
+ */
+ private void appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ }
+ }
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+ }
+ }
+
+ /**
+ * Pushes any partially-filled accumulator into its queue at row-group end so the emit path can
+ * dequeue it as the row group's final batch.
+ */
+ private void finalizePartialAccumulators() {
+ if (currentKeyAccumulatorRowCount == 0) return;
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Closes anything held by the splicing path: pending dequeued key vectors not yet rolled over,
+ * any vectors still queued (e.g. on early termination), and partially-filled accumulators.
+ * Called from {@link #close()}.
+ */
+ private void closeSplicingState() {
+ if (pendingCloseKeyVectors != null) {
+ for (WritableColumnVector v : pendingCloseKeyVectors) {
+ if (v != null) v.close();
+ }
+ pendingCloseKeyVectors = null;
+ }
+ if (keyVectorQueues != null) {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ if (q != null) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ }
+ }
+ if (currentKeyAccumulators != null) {
+ for (WritableColumnVector v : currentKeyAccumulators) {
+ if (v != null) v.close();
+ }
+ currentKeyAccumulators = null;
+ }
+ }
+
+ /**
+ * Per-key-column value copier: appends one value from {@code src[srcRow]} to
+ * {@code dst[dstRow]}. Picked once at init via {@link #copierFor(DataType)}; called per surviving
+ * row in {@link #appendSurvivorRowToAccumulators}. Caller handles null sources.
+ */
+ @FunctionalInterface
+ private interface ValueCopier {
+ void copy(WritableColumnVector dst, int dstRow, WritableColumnVector src, int srcRow);
+ }
+
+ /**
+ * Returns a {@link ValueCopier} for the given key {@link DataType}. The set of types handled here
+ * is the definition behind {@code ParquetStorageFilter.isSupportedKeyType}, which gates both
+ * planning-time extraction and {@code ParquetStorageFilter.create} -- so the throw at the end is
+ * unreachable. Teach both sides at once when adding a type; a type admitted there but missing
+ * here becomes a task failure instead of a planning-time rejection.
+ */
+ private static ValueCopier copierFor(DataType dt) {
+ if (dt instanceof BooleanType) {
+ return (dst, dRow, src, sRow) -> dst.putBoolean(dRow, src.getBoolean(sRow));
+ }
+ if (dt instanceof ByteType) {
+ return (dst, dRow, src, sRow) -> dst.putByte(dRow, src.getByte(sRow));
+ }
+ if (dt instanceof ShortType) {
+ return (dst, dRow, src, sRow) -> dst.putShort(dRow, src.getShort(sRow));
+ }
+ if (dt instanceof IntegerType
+ || dt instanceof DateType
+ || dt instanceof YearMonthIntervalType) {
+ return (dst, dRow, src, sRow) -> dst.putInt(dRow, src.getInt(sRow));
+ }
+ if (dt instanceof LongType
+ || dt instanceof TimestampType
+ || dt instanceof TimestampNTZType
+ || dt instanceof TimeType
+ || dt instanceof DayTimeIntervalType) {
+ return (dst, dRow, src, sRow) -> dst.putLong(dRow, src.getLong(sRow));
+ }
+ if (dt instanceof FloatType) {
+ return (dst, dRow, src, sRow) -> dst.putFloat(dRow, src.getFloat(sRow));
+ }
+ if (dt instanceof DoubleType) {
+ return (dst, dRow, src, sRow) -> dst.putDouble(dRow, src.getDouble(sRow));
+ }
+ if (dt instanceof DecimalType decimalType) {
+ int precision = decimalType.precision();
+ if (precision <= Decimal.MAX_INT_DIGITS()) {
+ return (dst, dRow, src, sRow) -> dst.putInt(dRow, src.getInt(sRow));
+ }
+ if (precision <= Decimal.MAX_LONG_DIGITS()) {
+ return (dst, dRow, src, sRow) -> dst.putLong(dRow, src.getLong(sRow));
+ }
+ return (dst, dRow, src, sRow) -> dst.putByteArray(dRow, src.getBinary(sRow));
+ }
+ if (dt instanceof StringType
+ || dt instanceof VarcharType
+ || dt instanceof CharType
+ || dt instanceof BinaryType) {
+ return (dst, dRow, src, sRow) -> dst.putByteArray(dRow, src.getBinary(sRow));
Review Comment:
**One throwaway `byte[]` per surviving row, in the innermost loop, for the most common key shape.**
`WritableColumnVector.getBinary` (L527) does `arrayData().getBytes(off, len)` -> `new byte[count]` + `System.arraycopy`; `putByteArray` -> `appendBytes` then copies again. So every survivor costs one garbage array plus two copies.
For a 1M-row row group at 50% selectivity with a 20-byte string join key that is 500K `byte[20]` allocations (~20MB of churn including headers) and ~20MB of redundant copying -- per row group, per key column. Dictionary-encoded pages skip the first allocation but still pay the expansion copy.
A vector-to-vector `appendBytes(int length, WritableColumnVector src, int srcOffset)` on `WritableColumnVector` -- one `System.arraycopy`/`Platform.copyMemory` between the two `arrayData()` buffers, zero allocation -- removes it. The `ByteBuffer` overload at `WritableColumnVector.java:449` is the existing precedent.
(This disappears entirely if phase 2 reads the key columns; see the review comment.)
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1885,6 +1885,21 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED =
+ buildConf("spark.sql.parquet.storageFilterPushdown.enabled")
+ .doc("If true, allows the vectorized Parquet reader to evaluate runtime storage filters " +
+ "(e.g. bloom filters from join runtime filtering) at the scan level using late " +
+ "materialization: read key columns first, evaluate the filter per row, then read data " +
+ "columns restricted to surviving rows. This is a planning-time decision only: when " +
+ "false, no storage filter is attached to a scan in the first place and the filter is " +
+ "applied as an ordinary post-scan filter instead. Note that the surviving key values of " +
+ "a whole row group are buffered before the first batch of that row group is produced, " +
+ "so a task holds up to one extra copy of the key columns for one row group.")
+ .version("5.0.0")
Review Comment:
Should this be `4.4.0`?
`dev/next_version_candidates.py` currently prints `master 5.0.0 / branch-4.x 4.4.0`, and `branch-4.x` is at `4.4.0-SNAPSHOT` and actively taking backports. This is an additive, opt-in, default-`false` feature, and the one public type whose signature changed (`FileSourceScanExec`, a new defaulted parameter) is under `org.apache.spark.sql.execution.*`, which `project/MimaExcludes.scala:171` excludes permanently -- so neither master-only exception (binary-incompatible, or a non-critical dependency bump) seems to apply.
Unless you are deliberately planning this as master-only, `5.0.0` will make `docs/sql-configuration` and `SHOW` claim the config arrived a full major later than it did.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +929,450 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+
+ // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the
+ // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and
+ // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes
+ // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index
+ // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's
+ // ranges under the full schema, and with it off we do not call getRowRanges at all.
+ lateMatReader.setRequestedSchema(requestedSchema);
+ // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT
+ // options.useColumnIndexFilter(), so calling it unconditionally would keep applying
+ // column-index filtering after a user turned it off. That conf is the documented escape hatch
+ // for files whose column index is wrong, and trusting a wrong column index here would drop
+ // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no
+ // longer holds this predicate.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ if (pushedFilterRanges.rowCount() == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // Both byte metrics are always wired in production (FileSourceScanLike creates all five
+ // whenever storageFilters is non-empty), so this only skips the work on the test-only path
+ // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so
+ // there is nothing here to avoid on the production path.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ long baselineRows = pushedFilterRanges.rowCount();
+ long baselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema,
+ pushedFilterRanges)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyRequestedSchema);
+ long phase1Bytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema,
+ pushedFilterRanges)
+ : 0L;
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite "
+ + pushedFilterRanges.rowCount() + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
Review Comment:
**This is where `LIMIT` short-circuiting is lost.**
`evaluateStorageFilter` walks the whole row group before this method returns, so `SELECT ... LIMIT 5` must decode and buffer every key value of row group 0 -- 1M+ values at the default `parquet.block.size` -- before the first batch can be produced. The eager path returns after `capacity` rows.
The early-termination tests (`ParquetStorageFilterSuite.scala:1799`, `:1818`) all use `rowGroupSize = 256L`, so this is not currently exercised. Chunk-at-a-time evaluation would fix it together with the buffering above, at the cost of building `finalRanges` incrementally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4158022188
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. None counts what it cost. A row group
+ * whose filter was given up reads its key columns twice and counts nothing, so the counters can
+ * show a saving on a scan that read more than it would have with the feature off.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums those skipped row groups' rows, counting per block the rows
+ * that survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned. A row group that gave
+ * splicing up reads its key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+
+ /** These counters keyed for the scan, which carries them without naming any of them. */
+ def toMap: Map[String, SQLMetric] = Map(
+ StorageFilterMetrics.ROW_GROUPS_SKIPPED -> rowGroupsSkipped,
+ StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP -> rowsExcludedByRowGroup,
+ StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP -> rowsExcludedWithinRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP -> bytesAvoidedByRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING -> bytesAvoidedByPageFiltering)
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /** A fresh set of counters, with the labels the SQL UI shows. */
+ def create(sparkContext: SparkContext): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped =
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ rowsExcludedByRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ rowsExcludedWithinRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ bytesAvoidedByRowGroup = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ bytesAvoidedByPageFiltering = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made: the map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ require(boundExpression != null,
+ "a storage filter must only be evaluated on an executor, and this copy has released the " +
+ "expression it would build its predicate from")
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader evaluates
+ * the predicate without the conjuncts that precede it in the plan, so one that throws on a row an
+ * earlier conjunct would have rejected throws where a plain scan does not. That is handled where
+ * it arises. The reader gives the filter up for the row group and reads it plainly, and the
+ * post-scan `Filter` then evaluates every conjunct in its own order.
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while applying a storage filter that it must not absorb by
+ * giving the filter up. Two kinds are not the reader's to absorb, and both are asked the way the
+ * rest of Spark asks them:
+ * - a pending task kill, since the error may be the kill's interrupt surfacing through a UDF
+ * that wrapped it. `PythonRunner` reads a kill off the task context in the same way.
+ * - a fatal error in the cause chain, found by `Executor.isFatalError`.
+ *
+ * Either goes out wrapped, see [[mustPropagate]].
+ */
+ def rethrowIfMustPropagate(e: Throwable): Unit = {
+ if (isKilled || Executor.isFatalError(e, fatalErrorDepth - 1)) throw mustPropagate(e)
Review Comment:
Thanks, the count was wrong, and the fix is your first option. The fatal error itself goes out now, so it reaches the executor at the top, whatever `FileScanRDD` wraps around it. Whether a cause counts is decided as deep as the executor would look had a plain plan raised the error. For an evaluation error that is the full depth, since `FilterExec` raises it as it is. For a decoding error it is one level less, for the FAILED_READ_FILE that `attachFilePath` adds. A kill keeps its checked wrapper, which the executor reports as killed at any depth. The checks also run in the executor's own order now: a fatal error at the top goes out as it is, kill or no kill, then a pending kill, then a fatal error deeper in the chain. A new test pins both boundaries and that order.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Exception e) {
+ // A corrupt page whose decoding raises an exception shows here, since parquet reads a
+ // chunk's bytes up front but decodes a page only when it is read. Giving the row group up
+ // hands it to phase 2 whole, which reads the same pages in the same batches as a plain
+ // scan. So the page fails at the same batch, after the same rows, and `ignoreCorruptFiles`
+ // keeps what it keeps of a plain read. A codec that reports corruption as an Error, as
+ // hadoop-lzo's `InternalError` does, is not caught here and fails before the first batch.
+ giveUpOnError(e, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
Review Comment:
Thanks. I took your second option, and fixed the message.
- The per-type tests now also assert that no survivor copy failed and gave splicing up. So a copier that throws fails its type's test even though the rows stay right. I checked that by making the `decimal_int` copier write a long.
- The WARN no longer says a vector could not grow. It says buffering a surviving key value raised an error, and it carries the exception.
- The copy moved out of the predicate's `try`, so a failure there gives splicing up, never the filter.
The catch still takes any `RuntimeException`, on purpose. The copy reads every key, but the filter need not read every key on every row. `coalesce(a, b)` reads `b` only where `a` is null. So the copy can fail on a value no plain read decodes, such as a `b` past a damaged dictionary. Failing the task there would fail a query a plain read answers. Giving splicing up reads the key columns the way a plain read does, so that value fails the read only where a plain read fails. A new test covers that case.
The same `reserve` failure while phase 1 decodes gives the filter up for that row group, since phase 1 decodes into vectors only this reader allocates, and phase 2 then allocates what a plain read does.
A raw `OutOfMemoryError` from `newKeyVector` or from `getBinary` is fatal, and it goes to the executor the way a plain read's own would. The catch is not meant to cover it.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
Review Comment:
Done as you suggest. `ParquetStorageFilter.internalError` throws `SparkException.internalError`, and is typed for Java as a `RuntimeException` it never returns. The survivor queue, the store row count, the key ordinal and primitive checks, `ValueCopier.forType`, `ParquetReadState.forRead` and the predicate's driver-side guard all go through it. The constant's predicate is resolved outside its fallback now, so that guard fails there too. The tests that expected an `IllegalStateException` now assert an internal error that `shouldIgnoreCorruptFileException` rejects.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Exception e) {
+ // A corrupt page whose decoding raises an exception shows here, since parquet reads a
+ // chunk's bytes up front but decodes a page only when it is read. Giving the row group up
+ // hands it to phase 2 whole, which reads the same pages in the same batches as a plain
+ // scan. So the page fails at the same batch, after the same rows, and `ignoreCorruptFiles`
+ // keeps what it keeps of a plain read. A codec that reports corruption as an Error, as
+ // hadoop-lzo's `InternalError` does, is not caught here and fails before the first batch.
+ giveUpOnError(e, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // A survivor vector that cannot grow, which `WritableColumnVector.reserve` reports as
+ // a RuntimeException even when an OutOfMemoryError is behind it. The buffer is this
+ // reader's own and optional, so this gives splicing up rather than the filter, and
+ // the memory goes back at once. A plain read never allocates it.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ALLOCATION);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together
+ // here.
+ long rangeBytes = survivorRangeCount * ROW_RANGE_BYTES;
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ } catch (Exception e) {
+ // Fail open on any exception. The predicate runs here without the conjuncts before it in
+ // the plan, so it can throw on a row a plain scan never evaluates it on (see
+ // `ParquetStorageFilter.isSupportedStorageFilter`). Giving the filter up returns every
+ // row the pushed filter kept, and the post-scan Filter decides them in its own order.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK error
+ // (`timestamp_seconds` of a decimal), and a Hive UDF throws a checked SparkException.
+ // Rethrowing would fail a query a plain read answers, or drop the rest of the file
+ // silently under `ignoreCorruptFiles`. Only a task kill or a fatal error is rethrown.
+ giveUpOnError(e, ROW_GROUP_GIVEN_UP_ON_EVALUATION);
+ return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
Review Comment:
Added to the `maxSplicedRowGroupBytes` doc. Off heap the key values are native memory outside `spark.memory.offHeap.size`, so they come out of `spark.executor.memoryOverhead`. Its first sentence also says it limits an estimate now, not a maximum. The up-front `reserve` you mention is made only when it grows the child past its default allocation and its charge fits what the cap leaves, so it is no longer allocated past the check. The `MemoryConsumer` follow-up stays in the description.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Exception e) {
+ // A corrupt page whose decoding raises an exception shows here, since parquet reads a
+ // chunk's bytes up front but decodes a page only when it is read. Giving the row group up
+ // hands it to phase 2 whole, which reads the same pages in the same batches as a plain
+ // scan. So the page fails at the same batch, after the same rows, and `ignoreCorruptFiles`
+ // keeps what it keeps of a plain read. A codec that reports corruption as an Error, as
+ // hadoop-lzo's `InternalError` does, is not caught here and fails before the first batch.
+ giveUpOnError(e, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
Review Comment:
The doc says so now. For a file without a page index the filter gains only where a whole row group has no surviving row, a bloom's false positives included, and every other row group reads its key columns twice. How many row groups that is depends on their size, their distinct keys, the selectivity and the bloom's false positive rate. So declining the rest of the file after a few such row groups is a heuristic, and it belongs with the gating heuristic the description lists as a follow-up, which names this case now. The give-up counter follow-up is what would make the loss visible in the metrics.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
Review Comment:
Done, per block in every case. The latch is gone. A `MissingOffsetIndexException` reads only that row group again whole, and the next one is narrowed as usual. The WARN is reported once per split. The damaged-index test now also asserts that the row groups around the damaged one keep the storage filter's narrowing, which the latch fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4145451506
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled when the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ *
+ * <p>That happens only before the first row group is loaded. The plain path reads through
+ * parquet's own row-group cursor, and the phases below read row groups by index, which never
+ * advances it. So nulling this later would restart the read at the first row group.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch's key slots hold the head of each survivor queue, not the persistent key vectors, so
+ // closing that batch would leave those unreached. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
Review Comment:
Fixed the first way. `SpecificParquetRecordReaderBase` keeps the `ParquetReadOptions` it builds, in both `initialize` paths that build a file reader, and `pageIndexDisabled` is `!readOptions.useColumnIndexFilter()`. So `totalRowCount` and the rows phase 0 reads come from one answer.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled when the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ *
+ * <p>That happens only before the first row group is loaded. The plain path reads through
+ * parquet's own row-group cursor, and the phases below read row groups by index, which never
+ * advances it. So nulling this later would restart the read at the first row group.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch's key slots hold the head of each survivor queue, not the persistent key vectors, so
+ // closing that batch would leave those unreached. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, and the plain read path skips them too. Past here one
+ // would reach `pushedFilterRangesFor`, whose `RowRanges.createSingle(0)` under
+ // `pageIndexDisabled` builds Range(0, -1) and trips its `from <= to` assertion.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Neither depends on the columns
+ // requested, so giving the filter up would only meet the same null in phase 2.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ long keptRows = dataPages.getRowCount();
Review Comment:
Added. Both reads assert that the store holds as many rows as were asked for, `keyPages` against the pushed filter's count and `dataPages` against `finalRowCount`, and fail with an `IllegalStateException` naming both numbers. With that in place `keptRows` is gone and the loop uses `finalRowCount`, which goes with your item on `installRowGroup` below.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled when the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ *
+ * <p>That happens only before the first row group is loaded. The plain path reads through
+ * parquet's own row-group cursor, and the phases below read row groups by index, which never
+ * advances it. So nulling this later would restart the read at the first row group.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch's key slots hold the head of each survivor queue, not the persistent key vectors, so
+ // closing that batch would leave those unreached. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, and the plain read path skips them too. Past here one
+ // would reach `pushedFilterRangesFor`, whose `RowRanges.createSingle(0)` under
+ // `pageIndexDisabled` builds Range(0, -1) and trips its `from <= to` assertion.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Neither depends on the columns
+ // requested, so giving the filter up would only meet the same null in phase 2.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ long keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // The same ranges as the baseline when nothing was pruned, and then the walk would arrive
+ // at the same number a second time.
+ long phase2Bytes = finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += blockAccounting.compressedBytesForRowRanges(blockIdx,
+ keyOnlyPaths, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ long filteredRows = baselineRows - keptRows;
+ if (filteredRows > 0) m.rowsExcludedWithinRowGroup().add(filteredRows);
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ RowRanges readerRanges = readerRangesFor(finalRanges, finalRowCount, blockRowCount);
+ installRowGroup(dataPages, readerRanges, keptRows);
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queues at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ return spliceCurrentRowGroup && isKeyTopLevel[slot];
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses, since the rows its column index already excluded were never this filter's
+ // to save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ *
+ * <p>A file this reader can no longer read in part still comes through here, since these are the
+ * rows a plain read of the same query would read, and leaving them to the post-scan Filter
+ * instead would make the scan slower than having no storage filter at all.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * What the surviving rows of the current row group cost to hold as row ranges, which is the half
+ * of the budget that a row group buffering nothing still pays. A filter whose survivors are
+ * scattered makes one range per surviving row, and phase 2 needs the whole set to select its
+ * pages. One set is held, not one per column reader, and the readers of a row group walk that
+ * same list, each with a cursor of its own.
+ */
+ private static long rowRangeStateBytes(long rangeCount) {
+ // Parquet's own `RowRanges.Range` holds two longs, plus its object header and a list slot.
+ return rangeCount * 40L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group. The
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ ensureKeyScratchAllocated();
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below, which is the bytes buffered
+ // for splicing and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so whether anything survives
+ // at all was the only thing left to learn, and it does. Stop here, since the ranges are
+ // not needed and the rest of the key column need not be decoded or evaluated. The
+ // row groups the filter empties are still skipped, which is the case that returns a
+ // non-null empty range set below.
+ abandonSplicing();
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) splicedBytes += appendSurvivorRowToAccumulators(r);
+ // Both halves of what this row group retains grow per survivor, and either can cross
+ // the budget on its own, so they are weighed together here and nowhere else. The cheaper
+ // concession comes first, which is to release the buffer, and the filter goes too if the
+ // ranges alone still do not fit.
+ long rangeBytes = rowRangeStateBytes(survivorRangeCount);
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper concession first, releasing the buffer, which is a no-op for a row group
+ // that has already given splicing up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ if (rangeBytes > cap) {
+ // The ranges being built are about to be thrown away, so stop evaluating the rest.
+ return null;
+ }
+ }
+ } catch (Exception e) {
+ // Fail open, whatever the exception is and wherever in the row it arose. The predicate
+ // ran on a row that, in the plan, an earlier conjunct would have rejected before it, so a
+ // plain scan never evaluates it there. Giving the filter up for this row group puts every
+ // row the pushed filter kept back in the output, and the post-scan Filter then evaluates
+ // the conjuncts in their own order. Either an earlier one drops the row before this
+ // expression runs, or it does not and the query fails the way it would have without this
+ // feature.
+ //
+ // Not narrowed to Spark's own errors, because a built-in expression can raise a plain JDK
+ // one, as `timestamp_seconds` of a decimal throws ArithmeticException from
+ // longValueExact. Nor to unchecked ones, because a Hive UDF in the key throws a checked
+ // SparkException (FAILED_EXECUTE_UDF) that nothing declares. Rethrowing has two bad ends,
+ // and neither is the error the plan would have raised. The query fails where a plain read
+ // succeeds, and under `ignoreCorruptFiles` this reader's exception is read as a corrupt
+ // file, which drops the rest of it silently.
+ //
+ // Copying a surviving key lands here too, which a survivor vector that cannot grow does,
+ // and reading the row group with nothing buffered is the answer to that as well.
+ logFilterGivenUpOnError(e);
+ abandonSplicing();
+ return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // Any partially-filled accumulator goes to its queue, so the emit path can dequeue it as this
+ // row group's final batch.
+ if (spliceCurrentRowGroup && currentKeyAccumulatorRowCount > 0) {
+ pushAccumulatorsToQueues();
+ }
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // The row the predicate evaluates holds the key columns phase 1 reads, and for a key column
+ // this file does not have, the constant vector `initBatch` built for it. `reset()` is a no-op
+ // on those, and this reader never closes them, since the emitted batch owns them.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors as the emitted batch holds them, in the order the predicate's row
+ * reads them. Which is what a missing key column reads as, and what phase 1 overwrites for a key
+ * column this file has. Taking a present key's batch vector and discarding it is a getter over a
+ * final field, and it saves holding the missing positions as state.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = columnVectors[keyIndices[p]].getValueVector();
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows
+ * each.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ currentKeyAccumulators[i] = newKeyVector(i);
+ }
+ }
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, handing them to their
+ * queues first if the last row filled them. All key columns advance in lockstep, which is what
+ * keeps the queues aligned. Returns the bytes the row added, which the caller weighs against its
+ * budget.
+ */
+ private long appendSurvivorRowToAccumulators(int srcRow) {
+ if (currentKeyAccumulatorRowCount == capacity) {
+ // Here rather than after the append that fills them, so that a row group whose survivors are
+ // a multiple of the capacity leaves no unused set of accumulators behind.
+ pushAccumulatorsToQueues();
+ ensureCurrentKeyAccumulatorsAllocated();
+ }
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final KeyColumn[] keys = keyColumns;
+ long valueBytes = 0L;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ KeyColumn key = keys[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ key.copier().copy(dst, dstRow, src, srcRow);
+ // Measured on the destination, because a dictionary-encoded source has no length of its
+ // own, its values being read through the dictionary.
+ if (key.variableLength()) {
+ valueBytes += VARIABLE_LENGTH_BYTES_FACTOR * dst.getArrayLength(dstRow);
+ }
+ }
+ }
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ return keyFixedBytesPerRow + valueBytes;
+ }
+
+ /**
+ * Hands every accumulator to its queue, which is what the emit path dequeues from. The row count
+ * goes with them, since it describes what the accumulators hold and they now hold nothing.
+ */
+ private void pushAccumulatorsToQueues() {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /** Releases the pages phase 2 read for the row group just emitted, if any. */
+ private void closeDataPages() {
+ if (dataPages != null) {
+ dataPages.close();
+ dataPages = null;
+ }
+ }
+
+ /**
+ * Reports the first row group of this file whose filter could not be evaluated. Once per file,
+ * because a file whose values do that tends to do it again, and the row groups that follow are
+ * still filtered normally.
+ */
+ private void logFilterGivenUpOnError(Exception e) {
+ if (loggedFilterEvaluationError) return;
+ loggedFilterEvaluationError = true;
+ LOG.warn("Reading a row group of {} without the storage filter, because applying it to a row "
Review Comment:
Fixed. There are three messages now, one per way the filter is given up on an error: evaluating it on a row, which costs a row group, evaluating it on a file's constant key values, which costs the file, and decoding a key page, which costs a row group. Each says what it cost. Its javadoc says the report is once per reader, which is once per split.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2755 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.SparkException
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = readerFn(file).asInstanceOf[Iterator[Object]].map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter =
+ keyTypeAtLeastFilter(Literal(threshold), LongType, metrics)
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ // Reads a key-type fixture, returning the survivor keys and the reader. The `extract` function
+ // pulls one value at a time from the batch's key column.
+ private def readKeysAll[T](
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T,
+ capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter, (batch, i) => extract(batch.column(0), i),
+ capacity)
+
+ // Builds a `k >= threshold` storage filter bound to position 0 against a `(k, v)` schema whose
+ // key has the given type.
+ private def keyTypeAtLeastFilter(
+ threshold: Literal,
+ keyType: DataType,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, keyType, nullable = false), threshold)
+ val requested = StructType(Seq(
+ StructField("k", keyType, nullable = false), StructField("v", StringType, nullable = true)))
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("mixed: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ // Output is exact. VectorizedColumnReader uses PageReadStore.getRowIndexes (driven by
+ // our finalRanges) to skip rows within partial pages, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted has
+ // to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(195L))
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs the reader's own initialization, and it
+ // used to run that a second time on top. A reader that declined the filter on the first pass,
+ // here because every projected column is a key column, then reached the filter it had just
+ // released and failed with an NPE instead of reading plainly.
+ withTempDir { dir =>
+ val keys = (1L to 20L)
+ val path = writeKeyParquetFile(dir, keys)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(15L))),
+ requested)
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ Utils.tryWithResource(new LateMaterializationParquetRecordReader(false, 4096, filter)) {
+ reader =>
+ reader.initialize(split, context)
+ reader.initBatch(new StructType(), null)
+ val read = mutable.ArrayBuffer[Long]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ assert(read == keys, s"every row must come back unfiltered; got ${read.size}")
+ }
+ }
+ }
+
+ test("multi-batch emit: survivor count exceeds capacity") {
+ // Drive the reader at capacity = 16 with a row group of 71 survivors out of 100. Exercises:
+ // - The per-key-column queue holding multiple full-capacity vectors plus a partial tail.
+ // - The published queue head getting closed at the start of every subsequent emit.
+ // - The batch's key slots being rewritten ceil(71/16) = 5 times.
+ // The filter keeps most of the row group rather than all of it, so a reader that declined the
+ // filter altogether would return more rows than this asserts rather than passing.
+ withTempDir { dir =>
+ val keys = (1L to 100L)
+ // Big rowGroupSize so all 100 rows fit in one row group.
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 64 * 1024L)
+ val filter = keyTypeAtLeastFilter(Literal(30L), LongType)
+ val (result, reader) =
+ readKeysAll(path, filter, (vec, i) => vec.getLong(i), capacity = 16)
+ try {
+ assert(result == keys.filter(_ >= 30L).toSeq,
+ s"expected the surviving keys in order across multiple batches; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("int key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the IntegerType branch of ValueCopier.
+ withTempDir { dir =>
+ val keys = (1 to 100)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyTypeAtLeastFilter(Literal(90), IntegerType)
+ val (result, reader) = readKeysAll(path, filter, (vec, i) => vec.getInt(i))
+ try {
+ assert(result.toSet == keys.filter(_ >= 90).toSet,
+ s"expected int keys >= 90; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("string key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the StringType branch of ValueCopier (variable-length byte copy via putByteArray).
+ withTempDir { dir =>
+ val keys = (1 to 20).map(i => f"k$i%03d")
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyTypeAtLeastFilter(
+ Literal.create("k015", StringType), StringType)
+ val (result, reader) =
+ readKeysAll(path, filter, (vec, i) => vec.getUTF8String(i).toString)
+ try {
+ assert(result.toSet == keys.filter(_ >= "k015").toSet,
+ s"expected string keys >= 'k015'; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(0L))),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ Literal(bloomBytes(BloomFilter.create(10))),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // Serializes a [[BloomFilter]] to bytes suitable for a [[Literal]].
+ // The same serialization a runtime bloom's Literal carries, so these tests build theirs the way
+ // the planner does.
+ private def bloomBytes(bf: BloomFilter): Array[Byte] = BloomFilterAggregate.serialize(bf)
+
+ // Computes `XxHash64(v)` using the same seed Spark uses for runtime bloom filters.
+ private def xxHash64(v: Any, dt: DataType): Long = {
+ new XxHash64(Seq(Literal(v, dt))).eval(InternalRow.empty).asInstanceOf[Long]
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = StructType(Seq(
+ StructField("k", decimalType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = true)))
+ val key = ThrowsCheckedOn(BoundReference(0, LongType, nullable = false), bad = 7L)
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // Returns the FileSourceScanExec for a parquet read, with `storageFilters` attached. Goes through
+ // Spark's normal planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind, the SQL conf check,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def scanWithStorageFilter(
+ path: String,
+ keyName: String,
+ threshold: Long): FileSourceScanExec = {
+ val df = spark.read.parquet(path).select("k", "v")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ val keyAttr = scan.output.find(_.name == keyName).getOrElse(fail(s"No $keyName in scan output"))
+ val expr = GreaterThanOrEqual(keyAttr, Literal(threshold))
+ scan.copy(storageFilters = Seq(expr))
+ }
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect().map(r => (r.getLong(0), r.getString(1)))
+ }
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 195L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sortBy(_._1)}; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. Phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ assert(scan.simpleString(200).contains("GreaterThan(v,"),
+ s"the data filter must actually be pushed for this test to mean anything: " +
+ scan.simpleString(200))
+ // Storage filter on `k` (key column).
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(
+ storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(100L))))
+
+ val collected = executePlanCollect(withSF).toSet
+ // Every row satisfies v > 'v_000', so the storage filter alone decides the result.
+ val expected = rows.filter(_._1 >= 100L).toSet
+ assert(collected == expected,
+ s"got ${collected.toSeq.sortBy(_._1)}; expected ${expected.toSeq.sortBy(_._1)}")
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms >= 1,
+ s"expected >= 1 bloom filter on scan.storageFilters; got $storageBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSet == Set(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves bloom filter as post-scan FilterExec when conf is off") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters when conf is off; got $storageBlooms")
+ assert(postScanBlooms >= 1,
+ s"expected bloom in post-scan FilterExec when conf is off; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("the feature still engages when ignoreCorruptFiles is on") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSet == Set((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since the
+ // post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty).getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+
+ test("a row group over the splice cap is read the plain way, same rows, more bytes") {
+ // Past the cap a row group is read the plain way. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Two things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. And
+ // the fallback must actually have happened, which is observable. It reads the key column a
+ // second time, so it transfers strictly more bytes. Without that second assertion the test
+ // would pass even if the cap never reached the reader.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group read the plain way past the cap still
+ // has to emit in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") {
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, fileSchema, new StructType(), fileSchema, Nil, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), countingHadoopConf(), metricMap())
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+ readCounting(path, readerFn)
+ }
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the
+ // row ranges. `k >= 350` keeps a contiguous run, so there is one range to hold.
+ val (plainRows, plainBytes) = run("100b")
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true))),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity = 16)
+ try {
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected a row group with no survivor at all; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group past the cap charges its second key read against the byte metric") {
+ // Giving splicing up means phase 2 reads the key columns a second time, while the baseline
+ // counts them once, in phase 1. That extra read is a cost against the saving, so the same file
+ // and filter must report less avoided than they do while splicing. Without that term the metric
+ // would credit the feature with bytes it did transfer.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+
+ def avoidedBytes(cap: Long, threshold: Long): Long = {
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))),
+ fileSchema, m, cap)
+ val (_, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => b.column(0).getLong(i), capacity = 16)
+ try m.bytesAvoidedByPageFiltering.value finally reader.close()
+ }
+
+ val spliced = avoidedBytes(64L * 1024 * 1024, 350L)
+ val plain = avoidedBytes(100L, 350L)
+ assert(spliced > 0, s"page filtering has to avoid something here; got $spliced")
+ assert(plain < spliced,
+ s"the second key read must count against the saving; plain=$plain spliced=$spliced")
+
+ // The two halves of the budget are weighed together, and this is the case that says so. Eight
+ // survivors of a long key buffer 8 * (1 + 8) = 72 bytes and fall in one range of 40, so
+ // neither half reaches the 100-byte cap on its own while the sum passes it at the seventh.
+ // Weighing them separately would keep splicing here, and report the larger saving for it.
+ val eightSpliced = avoidedBytes(64L * 1024 * 1024, 393L)
+ val eightPlain = avoidedBytes(100L, 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+ }
+ }
+
+ // Writes a parquet file the low-level way, with no offset index for any column.
+ private def writeParquetFileWithoutOffsetIndex(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]]): String =
+ writeParquetFileByHand(dir, blocks, (_, _) => false)
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The three row groups are ordered so that
+ // the file's first partial read is what discovers the missing index, and the two row groups
+ // after that discovery still get phase 1. The middle one is emptied by the filter and skipped
+ // whole, the last keeps rows and is read whole. The missing index is learned from the read that
+ // fails, so the first row group is the one that discovers it. Spark's own writer always writes
+ // the page index, hence the hand-built file.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileWithoutOffsetIndex(dir, blocks)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", LongType, nullable = false)))
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ // `withLogAppender` returns Unit, so the read's result is captured out of the block.
+ val logAppender = new LogAppender("no offset index")
+ var read: (Seq[(Long, Long)], VectorizedParquetRecordReader) = null
+ withLogAppender(logAppender) {
+ read = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ val (result, reader) = read
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering")),
+ "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index does not cost the file its page filtering") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, and the
+ // reader answers that by not trusting the page index for the rest of the file. Every row group
+ // after it would then be read whole, which is more than a plain read of the same query, since
+ // that keeps the pushed data filter's page pruning.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", LongType, nullable = false)))
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(50L)))
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"),
+ spark.sessionState.newHadoopConf(), metricMap())
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ val logAppender = new LogAppender("no offset index")
+ var values: Seq[Long] = null
+ withLogAppender(logAppender) {
+ values = readerFn(file).asInstanceOf[Iterator[Object]].flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(1).getLong)
+ case row: InternalRow => Seq(row.getLong(1))
+ }.toSeq
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering")),
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning, which is the regression this pins. Read whole they
+ // would contribute 200 rows rather than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(ref.getOffset)
+ raf.write(Array.fill(ref.getLength)(0xFF.toByte))
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry leaves the reader unable to narrow the rest of this file, since the next row
+ // group's store may be as unreadable as this one's. What it must not give up is the pushed data
+ // filter's own page pruning, which parquet did before the reader saw the block, and which a
+ // plain read of the same query keeps. Dropping that would read more than having no storage
+ // filter at all.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", LongType, nullable = false)))
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] = {
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"),
+ spark.sessionState.newHadoopConf(), metricMap())
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ readerFn(file).asInstanceOf[Iterator[Object]].flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(0).getLong)
+ case row: InternalRow => Seq(row.getLong(0))
+ }.toSeq
+ }
+ val logAppender = new LogAppender("damaged offset index")
+ var filtered: Seq[Long] = null
+ withLogAppender(logAppender) {
+ filtered = read(evenKeys)
+ }
+ assert(logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering")),
+ "the read must have discovered the damaged index, or this test proves nothing")
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry has latched.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the splice queues and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString), capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // The capacity decides where the budget is crossed. At 4096 the 200 survivors never fill an
+ // accumulator, so the ranges alone cross it; at 16 the buffer has been through several
+ // accumulators by then. Both have to abandon the buffer and give the filter up together.
+ // Keeping one without the other would leave phase 2 reading non-key columns while emit
+ // spliced keys from a queue holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true))),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString))
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSet == Set("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(42L, LongType))
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does. The reader never calls `initialize(partitionIndex)`, so it could not evaluate
+ // the expression, while the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for. A dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`, since `/` is floating-point division in Spark SQL, which would hand every
+ // key expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize, dictionary = dictionary)
+ }
+
+ // Every column chunk of the file, from its footer. The facts the tests below assert about a
+ // fixture are read off these rather than assumed from the write options.
+ private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] =
+ footerOf(filePath).getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq
+
+ private def footerOf(filePath: String): ParquetMetadata = {
+ val conf = spark.sessionState.newHadoopConf()
+ ParquetFooterReader.readFooter(
+ HadoopInputFile.fromPath(new Path(filePath), conf), ParquetMetadataConverter.NO_FILTER)
+ }
+
+ // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String, column: String = "k"): Set[Encoding] =
+ footerChunks(filePath).filter(_.getPath.toDotString == column)
+ .flatMap(_.getEncodings.asScala).toSet
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // builds the plain vectorized reader rather than the late-materialization one. Every read helper
+ // in this suite goes through here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need. The caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ if (storageFilter == null) {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } else {
+ new LateMaterializationParquetRecordReader(useOffHeap, capacity, storageFilter)
+ }
+ } { reader =>
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // A key value as a string in its *internal* representation, so the splicing path and the plain
+ // path can be compared without knowing the type and without external type conversion.
+ // `InternalRow.getAccessor` does the per-type read, which is why this takes the batch's row
+ // rather than the column vector.
+ private def renderValue(row: InternalRow, ordinal: Int, dt: DataType): String =
+ InternalRow.getAccessor(dt)(row, ordinal) match {
+ case null => "null"
+ // Two equal arrays are two objects, so the default toString would compare identities.
+ case bytes: Array[Byte] => bytes.mkString(",")
+ case value => value.toString
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path. Whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.getRow(i), 0, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage, one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why. There is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ } {
+ val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded"
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = StructType(Seq(
+ StructField("k", dt, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // This is a regression test. TimeType is an AtomicType that passes every planning-time gate
+ // (it is batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a
+ // TIME join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = StructType(Seq(
+ StructField("k", dt, nullable = true), StructField("v", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // What FileSourceStrategy offers, what ParquetStorageFilter.create admits and what the reader
+ // can copy are one list, because isSupportedKeyType answers from ValueCopier. Both are asserted
+ // here. A type the copier has no case for would be a task failure rather than a planning-time
+ // rejection, and the delegation is what rules that out.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ assert(ValueCopier.forType(dt) != null, s"$dt should have a value copier")
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ intercept[IllegalStateException](ValueCopier.forType(dt))
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = true), StructField("v", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("two key columns: both accumulators stay aligned with each other and with the data column") {
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300 => a in [100, 150]
+ val bound = And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)),
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1), "both key columns should be recognized")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, and the emit path depends on
+ // that. It pairs the k-th key slot of the batch with key-row position k, which is the identity
+ // only while the list is ascending. Nothing else in the suite supplies an unsorted one, because
+ // every other fixture happens to mention its keys in column order. Production does not promise
+ // that. The conjuncts arrive in `afterScanFilters` order, which follows the join. Without the
+ // sort this returns `a` and `b` swapped, with no exception to notice.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // Same predicate as the test above, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // The emit path rewrites the key slots alone, and a partition slot is not one of them. It sits
+ // past the data columns and must keep the constant vector `initBatch` populated for it.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite. OffHeapColumnVector.close() frees
+ // the native buffer, so a read after close is a crash rather than stale data. A second close
+ // is harmless, since close() zeroes the addresses it freed. capacity = 16 over 71 survivors
+ // forces 5 emits, each closing the previous emit's dequeued key vectors. The other 29 rows
+ // are what makes a declining reader fail this.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(30L))
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (30L to 100L),
+ s"expected the 71 surviving keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test(s"$mode vectors: row-group skipping and page filtering over a mixed projection") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(195L, m)
+ val (collected, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString),
+ useOffHeap = useOffHeap)
+ try {
+ assert(collected == rows.filter(_._1 >= 195L),
+ s"expected exact filtering; got ${collected.map(_._1)}")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and the
+ // partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // One ColumnarBatch is handed out for the whole read and its key slots are rewritten per
+ // batch, so a consumer holding on to an earlier getCurrentValue() would read the wrong
+ // vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new LateMaterializationParquetRecordReader(false, 8, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution, a column missing from the physical file -----
+
+ test("non-key column missing from a file: byte-avoided metrics tolerate a missing offset index") {
+ // ColumnIndexStore returns a null OffsetIndex for a column that is in the (clipped) requested
+ // schema but absent from the row group, which is exactly what schema evolution produces. The
+ // avoided-bytes walk runs over the full requested schema on every row group of every file, so
+ // without a null guard this is an NPE on the first row group of the older file.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // The older file has (k, v), the newer one (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect()
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSet
+ val expected = ((150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))).toSet
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // The avoided-bytes counters are what walk the offset index of the missing column, so a
+ // positive total is the evidence that the walk ran and coped. Asserting non-negativity
+ // would prove nothing. `SQLMetric.add` drops a negative and the value cannot go below zero.
+ val bytesRg = withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value + bytesPf.value > 0,
+ s"the walk must credit some avoided bytes; got rg=${bytesRg.value} pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = scanOf(spark.read.parquet(path).select("k", "v").queryExecution.executedPlan)
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Set[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Set[(Long, Option[Long])]) = {
+ val df = spark.sql(s"SELECT id, k FROM $table")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(buildFilter(keyAttr)))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = scan.output.indexWhere(_.name == "id")
+ val kPos = scan.output.indexWhere(_.name == "k")
+ val collected = rowPlan.executeCollect()
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSet
+ (withSF, collected)
+ }
+
+ test("all keys missing: an error on the constant reads the file rather than skipping it") {
+ // A file with none of the filter's key columns makes the predicate constant over it, so the
+ // reader decides once, from the values `initBatch` materialized for those columns. That
+ // evaluation can throw exactly as a row's value can. Here the older file's `k` reads as its
+ // DEFAULT, which is not a number, and the cast fails under ANSI. Failing open reads the file
+ // the way a plain scan would, so its rows survive. Skipping it would lose them, and rethrowing
+ // would fail a query a plain read answers.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true",
+ SQLConf.ANSI_ENABLED.key -> "true") {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql("ALTER TABLE evolved ADD COLUMN k STRING DEFAULT 'not-a-number'")
+ spark.sql("INSERT INTO evolved VALUES (4, '40'), (5, '50')")
+ val scan = scanOf(spark.sql("SELECT id, k FROM evolved").queryExecution.executedPlan)
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters =
+ Seq(GreaterThanOrEqual(Cast(keyAttr, LongType), Literal(1L))))
+ val idPos = scan.output.indexWhere(_.name == "id")
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val ids = rowPlan.executeCollect().map(_.getLong(idPos)).toSet
+ assert(ids == Set(1L, 2L, 3L, 4L, 5L),
+ s"the older file must fail open and the newer one pass the cast; got $ids")
+ }
+ }
+ }
+
+ test("all keys missing: a checked exception on the constant reads the file") {
+ // The constant evaluation fails open under the same rule as a row's, a checked exception
+ // included. The older file's `k` reads as its DEFAULT 7, which the key expression throws on,
+ // so that file must be read the way a plain scan would. The newer file's keys evaluate
+ // normally and are rejected, since no Filter runs above this scan.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(ThrowsCheckedOn(k, bad = 7L), Literal(100L)))
+ assert(collected == Set((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L))),
+ s"the older file must fail open and the newer one be filtered; got $collected")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test. The default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Set[(Long, Option[Long])] = Set((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(StorageFilterMetrics.ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Set((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got ${rejectsNull.toSeq.sorted}")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got ${acceptsNull.toSeq.sorted}")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect().map(r => (r.getLong(0), r.getLong(2))).toSet
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1)).toSet
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got " +
+ s"${collected.toSeq.sorted.take(5)}")
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select("k", "s")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(195L))))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect()
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSet
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k")).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sorted}; expected $expected")
+ }
+ }
+ }
+
+ test("a storage filter binds to the right column with partition and metadata columns in output") {
+ // `preparedStorageFilters` binds the filter's references against `output.take(requiredSchema
+ // .length)`, and `output` is the data columns, then the generated metadata columns, then the
+ // partition columns, then the constant metadata ones. A projection carrying all four is where a
+ // wrong prefix would bind the predicate to another column and drop rows the filter keeps, which
+ // the post-scan Filter cannot put back.
+ withTempDir { dir =>
+ val outDir = new File(dir, "parted").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "CONCAT('v_', id) AS v", "CAST(id % 3 AS INT) AS p")
+ .repartition(1)
+ .write
+ .partitionBy("p")
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select(
+ col("k"), col("v"), col("p"),
+ col("_metadata.row_index").as("ri"), col("_metadata.file_size").as("fs"))
+ val scan = scanOf(df.queryExecution.executedPlan)
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+ // The assumption the binding rests on, stated where a change to `output` would break it.
+ assert(withSF.output.take(withSF.requiredSchema.length).map(_.name) ==
+ withSF.requiredSchema.fieldNames.toSeq,
+ s"the scan's leading attributes must be its data columns; got " +
+ s"${withSF.output.map(_.name)}")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val pPos = withSF.output.indexWhere(_.name == "p")
+ def collectAll(plan: SparkPlan): Set[Seq[Any]] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ val types = plan.output.map(_.dataType)
+ rowPlan.executeCollect().map(_.toSeq(types)).toSet
+ }
+ val filtered = collectAll(withSF)
+ // The same scan without the filter is the oracle, so the row indexes and file sizes come
+ // from the read rather than from arithmetic over how the partitions were written.
+ val expected = collectAll(withSF.copy(storageFilters = Nil))
+ .filter(_(kPos).asInstanceOf[Long] >= 150L)
+ assert(filtered == expected,
+ s"expected ${expected.size} rows with their own partition and metadata values; " +
+ s"got ${filtered.size}")
+ assert(filtered.map(_(pPos)).size == 3,
+ s"every partition value must still be represented; got ${filtered.map(_(pPos))}")
+ }
+ }
+ }
+
+ // ----- Planner gates and the lost-filter invariant -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false. It may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ //
+ // This hook answers about shapes and types alone. Whether the conjunct is deterministic and
+ // whether its references are projected columns of this scan are plan-visible, so they stay with
+ // the planner; `FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter`
+ // covers that half.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ // A cast key is supported. In ANSI mode it can throw on a row an earlier conjunct would have
+ // dropped, and the reader handles that where it arises, by giving the filter up for the row
+ // group. Declining here instead would also decline every widening cast, which is what type
+ // coercion inserts for a join between an int and a bigint column.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+
+ // The conf is read in the format rather than in the planner, and it is the per-scan question
+ // that carries it, asked once before anything per conjunct.
+ assert(!format.supportsStorageFilterPushdown(spark), "the feature is off with the conf off")
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ assert(format.supportsStorageFilterPushdown(spark), "and on with it on")
+ assert(!subclass.supportsStorageFilterPushdown(spark), "a subclass must not claim support")
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ assert(format.supportsStorageFilter(onCast),
+ "a cast key is pushed, and an evaluation error gives the row group up")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The whole lost-filter safety argument rests on this gate. If the reader cannot do late
+ // materialization, the planner must NOT move the bloom out of the post-scan Filter.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one. Every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, "k", threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSet
+ assert(keys == (1L to 50L).toSet,
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support declines rather than delegating") {
+ // The default `FileFormat.buildReaderWithStorageFilters` answers None, and the caller falls
+ // back to the ordinary builder. That is what keeps the two builders from being able to call
+ // each other, and declining is safe because the post-scan Filter still holds the conjunct.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(1L)))
+ val declined = new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration(), Map.empty)
+ assert(declined.isEmpty, "the default must not build a reader of its own")
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilterPushdown(spark),
+ "and it must not claim support either")
+ }
+
+ test("a scan whose format declines falls back to the ordinary reader and returns every row") {
+ // The check above is that the default answers None. This is the other half, that `inputRDD`
+ // then builds the ordinary reader and the scan still produces its rows. A `ParquetFileFormat`
+ // subclass is the shape that reaches it, since it inherits the override and declines by class,
+ // and its ordinary builder is the one an extension would have customized.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanOf(spark.read.parquet(path).select("k", "v").queryExecution.executedPlan)
+ val keyAttr = scan.output.find(_.name == "k").get
+ val declining = scan.copy(
+ relation = scan.relation.copy(fileFormat = new ParquetFileFormat() {})(spark),
+ storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(195L))))
+ val collected = executePlanCollect(declining).toSet
+ assert(collected == rows.toSet,
+ s"a declining format reads the file plainly; got ${collected.size} of ${rows.size} rows")
+ }
+ }
+ }
+
+ Seq(false, true).foreach { aqe =>
+ test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") {
+ // AQE is on by default in production, and it is where the bloom subquery is planned by
+ // PlanAdaptiveSubqueries rather than PlanSubqueries, which is the path
+ // preparedStorageFilters' ScalarSubquery materialization depends on.
+ withBloomFilterTables {
+ val baseConf = Map(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString)
+ def run(pushdown: Boolean): (Int, Set[(Long, Long)]) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, rows) = runBloomFilterJoin()
+ (countBloomFiltersInStorageFilters(plan),
+ rows.map(r => (r.getLong(0), r.getLong(1))).toSet)
+ }
+ val (offFilters, off) = run(false)
+ val (onFilters, on) = run(true)
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ // Equal results are guaranteed by the post-scan Filter whether or not the scan applied the
+ // filter, so they alone would not notice AQE dropping it on the way to the final plan.
+ assert(offFilters == 0 && onFilters == 1,
+ s"the scan must carry the filter with the conf on and not with it off; " +
+ s"got on=$onFilters off=$offFilters")
+ }
+ }
+ }
+
+ test("canonicalization keeps a storage-filter scan distinct from a plain one") {
+ // storageFilters is in doCanonicalize and in the case class equality, which is what stops
+ // exchange and subquery reuse from serving one scan's result to the other. Reuse compares
+ // canonicalized plans, so a scan that filters must not match a plain scan of the same file, and
+ // two scans carrying the same filter must still match.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, "k", threshold = 25L)
+ val plain = withSF.copy(storageFilters = Nil)
+ assert(withSF != plain, "case class equality must take storageFilters into account")
+ assert(!withSF.sameResult(plain),
+ s"a filtering scan must not be reusable as a plain one:\n${withSF.canonicalized}\n" +
+ s"${plain.canonicalized}")
+ // A second, independently planned scan of the same file with the same filter still
+ // matches, so reuse is not disabled wholesale. Its key attribute carries a different
+ // exprId, which is what canonicalization normalizes away.
+ val sameSF = scanWithStorageFilter(path, "k", threshold = 25L)
+ assert(withSF.sameResult(sameSF),
+ s"two scans with the same storage filter must stay reusable:\n" +
+ s"${withSF.canonicalized}\n${sameSF.canonicalized}")
+ val otherSF = scanWithStorageFilter(path, "k", threshold = 30L)
+ assert(!withSF.sameResult(otherSF), "a different threshold is a different result")
+ }
+ }
+ }
+
+ test("whole-stage codegen off: the row-at-a-time path still applies the extracted bloom") {
Review Comment:
Fixed. The test now sums the two row-exclusion metrics of the scan that carries the storage filter, and asserts they are positive, with the same reason stated as in the `ignoreCorruptFiles` test.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2755 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.SparkException
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = readerFn(file).asInstanceOf[Iterator[Object]].map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter =
+ keyTypeAtLeastFilter(Literal(threshold), LongType, metrics)
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ // Reads a key-type fixture, returning the survivor keys and the reader. The `extract` function
+ // pulls one value at a time from the batch's key column.
+ private def readKeysAll[T](
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T,
+ capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter, (batch, i) => extract(batch.column(0), i),
+ capacity)
+
+ // Builds a `k >= threshold` storage filter bound to position 0 against a `(k, v)` schema whose
+ // key has the given type.
+ private def keyTypeAtLeastFilter(
+ threshold: Literal,
+ keyType: DataType,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, keyType, nullable = false), threshold)
+ val requested = StructType(Seq(
+ StructField("k", keyType, nullable = false), StructField("v", StringType, nullable = true)))
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("mixed: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ // Output is exact. VectorizedColumnReader uses PageReadStore.getRowIndexes (driven by
+ // our finalRanges) to skip rows within partial pages, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted has
+ // to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(195L))
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs the reader's own initialization, and it
+ // used to run that a second time on top. A reader that declined the filter on the first pass,
+ // here because every projected column is a key column, then reached the filter it had just
+ // released and failed with an NPE instead of reading plainly.
+ withTempDir { dir =>
+ val keys = (1L to 20L)
+ val path = writeKeyParquetFile(dir, keys)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(15L))),
+ requested)
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ Utils.tryWithResource(new LateMaterializationParquetRecordReader(false, 4096, filter)) {
+ reader =>
+ reader.initialize(split, context)
+ reader.initBatch(new StructType(), null)
+ val read = mutable.ArrayBuffer[Long]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ assert(read == keys, s"every row must come back unfiltered; got ${read.size}")
+ }
+ }
+ }
+
+ test("multi-batch emit: survivor count exceeds capacity") {
+ // Drive the reader at capacity = 16 with a row group of 71 survivors out of 100. Exercises:
+ // - The per-key-column queue holding multiple full-capacity vectors plus a partial tail.
+ // - The published queue head getting closed at the start of every subsequent emit.
+ // - The batch's key slots being rewritten ceil(71/16) = 5 times.
+ // The filter keeps most of the row group rather than all of it, so a reader that declined the
+ // filter altogether would return more rows than this asserts rather than passing.
+ withTempDir { dir =>
+ val keys = (1L to 100L)
+ // Big rowGroupSize so all 100 rows fit in one row group.
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 64 * 1024L)
+ val filter = keyTypeAtLeastFilter(Literal(30L), LongType)
+ val (result, reader) =
+ readKeysAll(path, filter, (vec, i) => vec.getLong(i), capacity = 16)
+ try {
+ assert(result == keys.filter(_ >= 30L).toSeq,
+ s"expected the surviving keys in order across multiple batches; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("int key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the IntegerType branch of ValueCopier.
+ withTempDir { dir =>
+ val keys = (1 to 100)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyTypeAtLeastFilter(Literal(90), IntegerType)
+ val (result, reader) = readKeysAll(path, filter, (vec, i) => vec.getInt(i))
+ try {
+ assert(result.toSet == keys.filter(_ >= 90).toSet,
+ s"expected int keys >= 90; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("string key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the StringType branch of ValueCopier (variable-length byte copy via putByteArray).
+ withTempDir { dir =>
+ val keys = (1 to 20).map(i => f"k$i%03d")
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyTypeAtLeastFilter(
+ Literal.create("k015", StringType), StringType)
+ val (result, reader) =
+ readKeysAll(path, filter, (vec, i) => vec.getUTF8String(i).toString)
+ try {
+ assert(result.toSet == keys.filter(_ >= "k015").toSet,
+ s"expected string keys >= 'k015'; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(0L))),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ Literal(bloomBytes(BloomFilter.create(10))),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // Serializes a [[BloomFilter]] to bytes suitable for a [[Literal]].
+ // The same serialization a runtime bloom's Literal carries, so these tests build theirs the way
+ // the planner does.
+ private def bloomBytes(bf: BloomFilter): Array[Byte] = BloomFilterAggregate.serialize(bf)
+
+ // Computes `XxHash64(v)` using the same seed Spark uses for runtime bloom filters.
+ private def xxHash64(v: Any, dt: DataType): Long = {
+ new XxHash64(Seq(Literal(v, dt))).eval(InternalRow.empty).asInstanceOf[Long]
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = StructType(Seq(
+ StructField("k", decimalType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = true)))
+ val key = ThrowsCheckedOn(BoundReference(0, LongType, nullable = false), bad = 7L)
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // Returns the FileSourceScanExec for a parquet read, with `storageFilters` attached. Goes through
+ // Spark's normal planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind, the SQL conf check,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def scanWithStorageFilter(
+ path: String,
+ keyName: String,
+ threshold: Long): FileSourceScanExec = {
+ val df = spark.read.parquet(path).select("k", "v")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ val keyAttr = scan.output.find(_.name == keyName).getOrElse(fail(s"No $keyName in scan output"))
+ val expr = GreaterThanOrEqual(keyAttr, Literal(threshold))
+ scan.copy(storageFilters = Seq(expr))
+ }
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect().map(r => (r.getLong(0), r.getString(1)))
+ }
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 195L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sortBy(_._1)}; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. Phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ assert(scan.simpleString(200).contains("GreaterThan(v,"),
+ s"the data filter must actually be pushed for this test to mean anything: " +
+ scan.simpleString(200))
+ // Storage filter on `k` (key column).
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(
+ storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(100L))))
+
+ val collected = executePlanCollect(withSF).toSet
+ // Every row satisfies v > 'v_000', so the storage filter alone decides the result.
+ val expected = rows.filter(_._1 >= 100L).toSet
+ assert(collected == expected,
+ s"got ${collected.toSeq.sortBy(_._1)}; expected ${expected.toSeq.sortBy(_._1)}")
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms >= 1,
+ s"expected >= 1 bloom filter on scan.storageFilters; got $storageBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSet == Set(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves bloom filter as post-scan FilterExec when conf is off") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters when conf is off; got $storageBlooms")
+ assert(postScanBlooms >= 1,
+ s"expected bloom in post-scan FilterExec when conf is off; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("the feature still engages when ignoreCorruptFiles is on") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSet == Set((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since the
+ // post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty).getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+
+ test("a row group over the splice cap is read the plain way, same rows, more bytes") {
+ // Past the cap a row group is read the plain way. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Two things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. And
+ // the fallback must actually have happened, which is observable. It reads the key column a
+ // second time, so it transfers strictly more bytes. Without that second assertion the test
+ // would pass even if the cap never reached the reader.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group read the plain way past the cap still
+ // has to emit in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") {
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, fileSchema, new StructType(), fileSchema, Nil, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), countingHadoopConf(), metricMap())
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+ readCounting(path, readerFn)
+ }
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the
+ // row ranges. `k >= 350` keeps a contiguous run, so there is one range to hold.
+ val (plainRows, plainBytes) = run("100b")
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true))),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity = 16)
+ try {
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected a row group with no survivor at all; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group past the cap charges its second key read against the byte metric") {
+ // Giving splicing up means phase 2 reads the key columns a second time, while the baseline
+ // counts them once, in phase 1. That extra read is a cost against the saving, so the same file
+ // and filter must report less avoided than they do while splicing. Without that term the metric
+ // would credit the feature with bytes it did transfer.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+
+ def avoidedBytes(cap: Long, threshold: Long): Long = {
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))),
+ fileSchema, m, cap)
+ val (_, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => b.column(0).getLong(i), capacity = 16)
+ try m.bytesAvoidedByPageFiltering.value finally reader.close()
+ }
+
+ val spliced = avoidedBytes(64L * 1024 * 1024, 350L)
+ val plain = avoidedBytes(100L, 350L)
+ assert(spliced > 0, s"page filtering has to avoid something here; got $spliced")
+ assert(plain < spliced,
+ s"the second key read must count against the saving; plain=$plain spliced=$spliced")
+
+ // The two halves of the budget are weighed together, and this is the case that says so. Eight
+ // survivors of a long key buffer 8 * (1 + 8) = 72 bytes and fall in one range of 40, so
+ // neither half reaches the 100-byte cap on its own while the sum passes it at the seventh.
+ // Weighing them separately would keep splicing here, and report the larger saving for it.
+ val eightSpliced = avoidedBytes(64L * 1024 * 1024, 393L)
+ val eightPlain = avoidedBytes(100L, 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+ }
+ }
+
+ // Writes a parquet file the low-level way, with no offset index for any column.
+ private def writeParquetFileWithoutOffsetIndex(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]]): String =
+ writeParquetFileByHand(dir, blocks, (_, _) => false)
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The three row groups are ordered so that
+ // the file's first partial read is what discovers the missing index, and the two row groups
+ // after that discovery still get phase 1. The middle one is emptied by the filter and skipped
+ // whole, the last keeps rows and is read whole. The missing index is learned from the read that
+ // fails, so the first row group is the one that discovers it. Spark's own writer always writes
+ // the page index, hence the hand-built file.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileWithoutOffsetIndex(dir, blocks)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", LongType, nullable = false)))
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ // `withLogAppender` returns Unit, so the read's result is captured out of the block.
+ val logAppender = new LogAppender("no offset index")
+ var read: (Seq[(Long, Long)], VectorizedParquetRecordReader) = null
+ withLogAppender(logAppender) {
+ read = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ val (result, reader) = read
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering")),
+ "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index does not cost the file its page filtering") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, and the
+ // reader answers that by not trusting the page index for the rest of the file. Every row group
+ // after it would then be read whole, which is more than a plain read of the same query, since
+ // that keeps the pushed data filter's page pruning.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", LongType, nullable = false)))
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(50L)))
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"),
+ spark.sessionState.newHadoopConf(), metricMap())
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ val logAppender = new LogAppender("no offset index")
+ var values: Seq[Long] = null
+ withLogAppender(logAppender) {
+ values = readerFn(file).asInstanceOf[Iterator[Object]].flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(1).getLong)
+ case row: InternalRow => Seq(row.getLong(1))
+ }.toSeq
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering")),
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning, which is the regression this pins. Read whole they
+ // would contribute 200 rows rather than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(ref.getOffset)
+ raf.write(Array.fill(ref.getLength)(0xFF.toByte))
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry leaves the reader unable to narrow the rest of this file, since the next row
+ // group's store may be as unreadable as this one's. What it must not give up is the pushed data
+ // filter's own page pruning, which parquet did before the reader saw the block, and which a
+ // plain read of the same query keeps. Dropping that would read more than having no storage
+ // filter at all.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", LongType, nullable = false)))
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] = {
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"),
+ spark.sessionState.newHadoopConf(), metricMap())
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ readerFn(file).asInstanceOf[Iterator[Object]].flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(0).getLong)
+ case row: InternalRow => Seq(row.getLong(0))
+ }.toSeq
+ }
+ val logAppender = new LogAppender("damaged offset index")
+ var filtered: Seq[Long] = null
+ withLogAppender(logAppender) {
+ filtered = read(evenKeys)
+ }
+ assert(logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering")),
+ "the read must have discovered the damaged index, or this test proves nothing")
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry has latched.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the splice queues and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString), capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // The capacity decides where the budget is crossed. At 4096 the 200 survivors never fill an
+ // accumulator, so the ranges alone cross it; at 16 the buffer has been through several
+ // accumulators by then. Both have to abandon the buffer and give the filter up together.
+ // Keeping one without the other would leave phase 2 reading non-key columns while emit
+ // spliced keys from a queue holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true))),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString))
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSet == Set("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(42L, LongType))
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does. The reader never calls `initialize(partitionIndex)`, so it could not evaluate
+ // the expression, while the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for. A dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`, since `/` is floating-point division in Spark SQL, which would hand every
+ // key expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize, dictionary = dictionary)
+ }
+
+ // Every column chunk of the file, from its footer. The facts the tests below assert about a
+ // fixture are read off these rather than assumed from the write options.
+ private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] =
+ footerOf(filePath).getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq
+
+ private def footerOf(filePath: String): ParquetMetadata = {
+ val conf = spark.sessionState.newHadoopConf()
+ ParquetFooterReader.readFooter(
+ HadoopInputFile.fromPath(new Path(filePath), conf), ParquetMetadataConverter.NO_FILTER)
+ }
+
+ // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String, column: String = "k"): Set[Encoding] =
+ footerChunks(filePath).filter(_.getPath.toDotString == column)
+ .flatMap(_.getEncodings.asScala).toSet
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // builds the plain vectorized reader rather than the late-materialization one. Every read helper
+ // in this suite goes through here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need. The caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ if (storageFilter == null) {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } else {
+ new LateMaterializationParquetRecordReader(useOffHeap, capacity, storageFilter)
+ }
+ } { reader =>
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // A key value as a string in its *internal* representation, so the splicing path and the plain
+ // path can be compared without knowing the type and without external type conversion.
+ // `InternalRow.getAccessor` does the per-type read, which is why this takes the batch's row
+ // rather than the column vector.
+ private def renderValue(row: InternalRow, ordinal: Int, dt: DataType): String =
+ InternalRow.getAccessor(dt)(row, ordinal) match {
+ case null => "null"
+ // Two equal arrays are two objects, so the default toString would compare identities.
+ case bytes: Array[Byte] => bytes.mkString(",")
+ case value => value.toString
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path. Whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.getRow(i), 0, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage, one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why. There is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ } {
+ val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded"
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = StructType(Seq(
+ StructField("k", dt, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // This is a regression test. TimeType is an AtomicType that passes every planning-time gate
+ // (it is batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a
+ // TIME join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = StructType(Seq(
+ StructField("k", dt, nullable = true), StructField("v", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // What FileSourceStrategy offers, what ParquetStorageFilter.create admits and what the reader
+ // can copy are one list, because isSupportedKeyType answers from ValueCopier. Both are asserted
+ // here. A type the copier has no case for would be a task failure rather than a planning-time
+ // rejection, and the delegation is what rules that out.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ assert(ValueCopier.forType(dt) != null, s"$dt should have a value copier")
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ intercept[IllegalStateException](ValueCopier.forType(dt))
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = true), StructField("v", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("two key columns: both accumulators stay aligned with each other and with the data column") {
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300 => a in [100, 150]
+ val bound = And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)),
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1), "both key columns should be recognized")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, and the emit path depends on
+ // that. It pairs the k-th key slot of the batch with key-row position k, which is the identity
+ // only while the list is ascending. Nothing else in the suite supplies an unsorted one, because
+ // every other fixture happens to mention its keys in column order. Production does not promise
+ // that. The conjuncts arrive in `afterScanFilters` order, which follows the join. Without the
+ // sort this returns `a` and `b` swapped, with no exception to notice.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // Same predicate as the test above, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // The emit path rewrites the key slots alone, and a partition slot is not one of them. It sits
+ // past the data columns and must keep the constant vector `initBatch` populated for it.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite. OffHeapColumnVector.close() frees
+ // the native buffer, so a read after close is a crash rather than stale data. A second close
+ // is harmless, since close() zeroes the addresses it freed. capacity = 16 over 71 survivors
+ // forces 5 emits, each closing the previous emit's dequeued key vectors. The other 29 rows
+ // are what makes a declining reader fail this.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(30L))
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (30L to 100L),
+ s"expected the 71 surviving keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test(s"$mode vectors: row-group skipping and page filtering over a mixed projection") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(195L, m)
+ val (collected, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString),
+ useOffHeap = useOffHeap)
+ try {
+ assert(collected == rows.filter(_._1 >= 195L),
+ s"expected exact filtering; got ${collected.map(_._1)}")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and the
+ // partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // One ColumnarBatch is handed out for the whole read and its key slots are rewritten per
+ // batch, so a consumer holding on to an earlier getCurrentValue() would read the wrong
+ // vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new LateMaterializationParquetRecordReader(false, 8, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution, a column missing from the physical file -----
+
+ test("non-key column missing from a file: byte-avoided metrics tolerate a missing offset index") {
Review Comment:
Renamed and rewritten. It is now "non-key column missing from a file: the evolved table reads and credits avoided bytes". The comments say what it checks: schema evolution end to end with the non-key column missing, the byte walks skipping it because they go over the block's chunks, and non-zero byte metrics. The `offsetIndex == null` branch stays on the description's knowingly-untested list, since it is an ordering bug rather than a file shape.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.SparkContext
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * added on top of a read of the same projection without one.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums those skipped row groups' rows, counting per block the rows
+ * that survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it, and it is zero on an
+ * all-keys projection, which can avoid nothing.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page. An all-keys projection has no page filtering at
Review Comment:
Removed from the metrics doc, in both places. The early-release comment in `loadNextRowGroup` now names a row group the filter empties and the end of the file. The description's follow-up no longer says an all-keys projection pays the overhead. It says that overhead is why neither the planner nor the reader pushes to one.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -476,34 +506,80 @@ private boolean containsPath(Type parquetType, String[] path, int depth) {
return false;
}
- private void checkEndOfRowGroup() throws IOException {
- if (rowsReturned != totalCountLoadedSoFar) return;
+ /**
+ * Loads the row group the read continues with, and returns false if there is none left. Only a
+ * subclass that picks its own rows can answer false, because this class reads every row of every
+ * row group, and then {@code totalRowCount} already says when they run out.
+ */
+ protected boolean loadNextRowGroup() throws IOException {
PageReadStore pages = reader.readNextRowGroup();
if (pages == null) {
throw new IOException("expecting more rows but reached last block. Read "
+ rowsReturned + " out of " + totalRowCount);
}
+ // Null ranges, because this path never picks the rows itself, and whether parquet narrowed the
+ // row group by a pushed filter's column index is known only to the store it handed back.
+ installRowGroup(pages, null, pages.getRowCount());
+ return true;
+ }
+
+ /**
+ * Points the column readers at one row group's {@code pages}, from which they are to read the
+ * rows {@code rowRanges} names, or every row of the row group when it is null. {@code rowCount}
Review Comment:
Fixed in all three places. `installRowGroup` says null ranges leave the rows to the store, which names them through its row indexes when parquet filtered it and otherwise holds every row. `newColumnReader` points there. `loadNextRowGroup` says this class reads every row group parquet hands it, not every row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4133528530
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -771,6 +836,28 @@ case class FileSourceScanExec(
readRDD
}
+ // Materialize scalar subqueries inside storage filters to literals and bind AttributeReferences
+ // to BoundReferences targeting positions in `requiredSchema`. Subqueries must have been prepared
+ // by SparkPlan before this is forced (same contract as `pushedDownFilters`).
+ @transient
+ protected lazy val preparedStorageFilters: Seq[Expression] = {
+ if (storageFilters.isEmpty) {
+ Nil
+ } else {
+ // No conf check here: the conf decides at planning time whether a scan is offered storage
+ // filters at all, and re-reading it now could only make this scan drop work it already has.
+ //
+ // `output` is `readDataColumns ++ generatedMetadataColumns ++ partitionColumns ++
+ // constantMetadataColumns` and `requiredSchema` is the StructType of the first two groups, so
+ // the first `requiredSchema.length` attributes line up with its fields.
+ val requestedDataAttrs = output.take(requiredSchema.length)
+ storageFilters.map { expr =>
+ val subqueryReplaced = expr.transform { case s: execution.ScalarSubquery => s.toLiteral }
+ BindReferences.bindReference(subqueryReplaced, requestedDataAttrs)
Review Comment:
The binding is documented now rather than carried twice. `buildReaderWithStorageFilters`'s scaladoc says the expressions arrive bound to `requiredSchema`, as `BoundReference`s over the row a reader assembles from it, which is the only form a reader can evaluate per row without resolving names again. `supportsStorageFilter` is asked about the unbound expression, where the names and the metadata are still there, so that is where a format decides. The two are index-aligned, one prepared filter per offered conjunct in the same order.
I did not add a second sequence or a pair type. It doubles a public `FileFormat` method's contract for a format that would then have to correlate the two itself, and the drift you name is the reason not to. If a format ever needs the original at build time, the aligned pair is the change to make then.
The fallback is covered end to end now. A scan whose relation carries a `ParquetFileFormat` subclass and a storage filter returns every row of the file through `buildReaderWithPartitionValues`, and the test fails if the subclass guard is removed.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1947,6 +1947,51 @@ object SQLConf {
.checkValue(threshold => threshold >= 0, "The threshold must not be negative.")
.createWithDefault(10)
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED =
+ buildConf("spark.sql.parquet.storageFilterPushdown.enabled")
+ .doc("If true, allows the vectorized Parquet reader to evaluate runtime storage filters " +
+ "(e.g. bloom filters from join runtime filtering) at the scan level using late " +
+ "materialization: read key columns first, evaluate the filter per row, then read data " +
+ "columns restricted to surviving rows. This is a planning-time decision only: when " +
+ "false, no storage filter is attached to a scan in the first place and the filter is " +
+ "applied as an ordinary post-scan filter alone. A pushed filter stays in the post-scan " +
+ "filter as well, the way a pushed data filter does, so honoring it is optional: a reader " +
+ "that meets a row group it cannot prune reads it the way a plain scan would, plus one " +
+ "more read of the key columns, since the phase that evaluated the filter already read " +
+ "them. A file written with no Parquet page index is read with the filter applied only " +
+ "where it empties a whole row group, since narrowing to part of one needs that index, so " +
+ "every row group of it that keeps a row pays that. Setting " +
+ "parquet.filter.columnindex.enabled to false turns this off entirely, because reading " +
+ "part of a row group goes through the page index. Note that " +
+ "the surviving key values of a whole row group are buffered before the " +
+ "first batch of that row group is produced, so a task holds up to one extra copy of the " +
+ "key columns for one row group.")
+ .version("5.0.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES =
+ buildConf("spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes")
+ .internal()
+ .doc("Most memory, in bytes, that the vectorized Parquet reader holds for one row group " +
Review Comment:
Rewritten. It opens with `The maximum memory, in bytes, that ...` and the accounting rules are separate sentences. The other conf doc was shortened in the same pass.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java:
##########
@@ -19,30 +19,29 @@
import org.apache.parquet.column.ColumnDescriptor;
-import java.util.ArrayList;
-import java.util.Iterator;
-import java.util.List;
import java.util.PrimitiveIterator;
/**
* Helper class to store intermediate state while reading a Parquet column chunk.
*/
final class ParquetReadState {
- /** A special row range used when there is no row indexes (hence all rows must be included) */
- private static final RowRange MAX_ROW_RANGE = new RowRange(Long.MIN_VALUE, Long.MAX_VALUE);
+ /** The row indexes to include, only not-null if the column index is present. */
Review Comment:
The comment is gone. `ParquetReadState` is a small class per way of saying which rows to include, and one factory picks between them. Its javadoc states the actual condition: ranges when the caller of the read holds them, which is this feature's phase 2, the page store's row indexes when only the store can describe them, which is parquet's own filtering, and neither, which means every row of the chunk.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.SparkThrowable
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Literal, Predicate, XxHash64}
+import org.apache.spark.sql.catalyst.trees.TreePattern.PLAN_EXPRESSION
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.{BinaryType, BooleanType, ByteType, DataType, DateType, DayTimeIntervalType, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType, StringType, StructType, TimestampNTZType, TimestampType, TimeType, YearMonthIntervalType}
+
+/**
+ * Optional SQL metrics the reader updates while applying a [[ParquetStorageFilter]]. Every
+ * counter is scoped to what the storage filter added on top of a read of the same projection
+ * without one. All fields are nullable; a null field disables that metric.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums the rows those skips excluded, per skipped block the rows that
+ * survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it, and it is zero on an
+ * all-keys projection, which can avoid nothing.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned.
+ *
+ * The row counters' suffix says where a row was excluded, not what would have saved it: a row
+ * inside a kept row group is read as part of its page and dropped during decode, so page
Review Comment:
Rewritten. The doc says the counters' suffix is where a row was excluded, not whether reading it was avoided, and it names both cases, a row read as part of a page that held a survivor and a row whose page phase 2 skipped. What a page-level saving was is what the byte counters say, and they are measured against the non-key bytes a plain read of the same projection would have transferred.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2526 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayOutputStream, File}
+import java.net.URI
+import java.time.LocalTime
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapreduce.Job
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName}
+import org.apache.parquet.hadoop.util.HadoopOutputFile
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, XxHash64}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FileSourceScanLike, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests the late-materialization path of [[VectorizedParquetRecordReader]] driven by a
+ * [[ParquetStorageFilter]]. Writes small multi-row-group parquet files, wires a hand-built filter
+ * into the reader, and asserts correctness + the two storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false), StructField("v", StringType, nullable = false)))
+ ParquetStorageFilter.create(Seq(expr), requested, metrics)
+ }
+
+ // Writes a single-column (just `k`) parquet file for the supplied key type via Spark's
+ // {@code Encoder}.
+ private def writeKeyOnlyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(dir, spark.createDataset(keys).toDF("k"), rowGroupSize)
+
+ // Reads a key-only file, returning the survivor keys and the reader. The {@code extract} function
+ // pulls one value at a time from the batch's key column.
+ private def readKeyOnlyAll[T](
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T,
+ capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k"), storageFilter, (batch, i) => extract(batch.column(0), i),
+ capacity)
+
+ // Builds a `k >= threshold` storage filter bound to position 0 against a key-only schema of the
+ // given key type.
+ private def keyOnlyAtLeastFilter(
+ threshold: Literal,
+ keyType: DataType,
+ metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, keyType, nullable = false), threshold)
+ val requested = StructType(Seq(StructField("k", keyType, nullable = false)))
+ ParquetStorageFilter.create(Seq(expr), requested, metrics)
+ }
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows in one row group: parquet's first row-group size check is at record 100, so
+ // `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = keyAtLeastFilter(1000L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf,
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(rgSkipped.value > 0,
+ s"expected at least one row group skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedRg.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${rowsExcludedRg.value}")
+ assert(rowsExcludedPf.value == 0,
+ s"no partial-row-group filtering expected; got ${rowsExcludedPf.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(bytesAvoidedRg.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got ${bytesAvoidedRg.value}")
+ assert(bytesAvoidedPf.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got ${bytesAvoidedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val filter = keyAtLeastFilter(0L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped, rowsExcludedWithinRowGroup = rowsExcludedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(rgSkipped.value == 0, s"nothing should be skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedPf.value == 0,
+ s"nothing should be filtered; got ${rowsExcludedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("mixed: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ // Output is exact: VectorizedColumnReader uses PageReadStore.getRowIndexes (driven by
+ // our finalRanges) to skip rows within partial pages, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it: a row group that was neither skipped whole nor emitted has
+ // to have its rows counted there.
+ assert(rowsExcludedPf.value > 0,
+ s"expected rows excluded inside a kept row group; got ${rowsExcludedPf.value}")
+ assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == rows.size,
+ s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " +
+ s"should account for all ${rows.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key-only projection: phase 2 is skipped and both byte-avoided metrics are zero") {
+ // When the projected schema contains only the bloom key, phase 2 is skipped entirely
+ // (`nonKeyColumns == null` in the reader). All output rows come from the per-key-column
+ // queues populated in phase 1. Total bytes read match the no-storage-filter path (phase 1 reads
+ // the key column once instead of phase 2 re-reading it), so both `avoided` metrics are zero:
+ // there are no non-key bytes to skip. This is the shape the design notes call the biggest win,
+ // so it must not be the shape that pays for metrics.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ // k >= 195 -> last 6 keys survive; preceding row groups skipped or page-pruned.
+ val filter = keyOnlyAtLeastFilter(Literal(195L), LongType, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf,
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i))
+ try {
+ val expected = keys.filter(_ >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact survivor keys; got ${result.sorted}, expected ${expected.toSeq.sorted}")
+ assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedRg.value > 0,
+ s"a skipped row group must count its rows too; got ${rowsExcludedRg.value}")
+ // The all-keys path takes its kept-row count from `finalRowCount` rather than from a
+ // phase-2 page store, so this identity is the only thing that checks that arithmetic.
+ assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == keys.size,
+ s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " +
+ s"should account for all ${keys.size} rows")
+ // For an all-keys projection, the no-storage-filter path would have read the same key
+ // column phase 1 reads. There are no non-key bytes to avoid; both metrics are 0.
+ assert(bytesAvoidedRg.value == 0,
+ s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedRg.value}")
+ assert(bytesAvoidedPf.value == 0,
+ s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("multi-batch emit: survivor count exceeds capacity") {
+ // Drive the reader at capacity = 16 with a row group of 100 surviving rows. Exercises:
+ // - The per-key-column queue holding multiple full-capacity vectors plus a partial tail.
+ // - The published queue head getting closed at the start of every subsequent emit.
+ // - The batch's key slots being rewritten ceil(100/16) = 7 times.
+ withTempDir { dir =>
+ val keys = (1L to 100L)
+ // Big rowGroupSize so all 100 rows fit in one row group.
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 64 * 1024L)
+ // Filter accepts every row so the queue is fully populated.
+ val filter = keyOnlyAtLeastFilter(Literal(0L), LongType)
+ val (result, reader) =
+ readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i), capacity = 16)
+ try {
+ assert(result == keys.toSeq,
+ s"expected all keys returned in order across multiple batches; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("int key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the IntegerType branch of ValueCopier.
+ withTempDir { dir =>
+ val keys = (1 to 100)
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyOnlyAtLeastFilter(Literal(90), IntegerType)
+ val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getInt(i))
+ try {
+ assert(result.toSet == keys.filter(_ >= 90).toSet,
+ s"expected int keys >= 90; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("string key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the StringType branch of ValueCopier (variable-length byte copy via putByteArray).
+ withTempDir { dir =>
+ val keys = (1 to 20).map(i => f"k$i%03d")
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyOnlyAtLeastFilter(
+ Literal.create("k015", StringType), StringType)
+ val (result, reader) =
+ readKeyOnlyAll(path, filter, (vec, i) => vec.getUTF8String(i).toString)
+ try {
+ assert(result.toSet == keys.filter(_ >= "k015").toSet,
+ s"expected string keys >= 'k015'; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction: storageFiltersFor pre-checks each one, and
+ // by the time create runs the conjunct is gone from the post-scan Filter, so a soft rejection
Review Comment:
Both rewritten. The metric test asserts all five counters, and the `create` test says the rows are safe either way because the conjunct stays in the post-scan `Filter`, which is what makes a hard rejection the right answer to a planner bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5972295466
@cloud-fan @sunchao, could you take a look when you have time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4149437730
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2757 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkException, TaskContext}
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading the
+ // rest of a file without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender("partial-read warning")
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val warned = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering"))
+ (result.get, warned)
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
Review Comment:
Changed. Every result check on a storage-filter scan compares an ordered or a sorted list now. The reader-level ones compare in file order, the planner-level ones sorted, and the schema-evolution helper returns a sorted `Seq`. The single-row query checks compare a `Seq` too.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
Review Comment:
Both fixed. `partialReadUnavailable` says the storage filter no longer narrows a row group for the rest of the split, while the pushed filter's own pruning still applies. The test comment says the same, per split. `recordFileSkipped` says it counts every row group the pushed data filter left rows in.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
+ * read for the rest of this split. The pushed data filter's own page pruning still applies, since
+ * parquet builds those ranges and hands back whole any block it has no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
Review Comment:
Rewritten. It says a consumer that stops between batches, such as a limit or the end of the task, leaves them for `close()` to free.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, TaskContext}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. None counts what it cost. A row group
+ * whose filter was given up reads its key columns twice and counts nothing, so the counters can
+ * show a saving on a scan that read more than it would have with the feature off.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums those skipped row groups' rows, counting per block the rows
+ * that survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned. A row group that gave
+ * splicing up reads its key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+
+ /** These counters keyed for the scan, which carries them without naming any of them. */
+ def toMap: Map[String, SQLMetric] = Map(
+ StorageFilterMetrics.ROW_GROUPS_SKIPPED -> rowGroupsSkipped,
+ StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP -> rowsExcludedByRowGroup,
+ StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP -> rowsExcludedWithinRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP -> bytesAvoidedByRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING -> bytesAvoidedByPageFiltering)
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /** A fresh set of counters, with the labels the SQL UI shows. */
+ def create(sparkContext: SparkContext): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped =
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ rowsExcludedByRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ rowsExcludedWithinRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ bytesAvoidedByRowGroup = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ bytesAvoidedByPageFiltering = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made: the map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer
+ // construction to first use on the executor.
+ @transient private lazy val predicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ require(boundExpression != null,
+ "a storage filter must only be evaluated on an executor, and this copy has released the " +
+ "expression it would build its predicate from")
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay.
+ */
+ def preparedPredicate: BasePredicate = predicate
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queues where the row group was spliced, from the overwritten vector where it
Review Comment:
Fixed in all four. It is "the survivor queue" everywhere, and the test is now "non-key column before the key column: each key's survivors go to its batch slot".
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
+ * read for the rest of this split. The pushed data filter's own page pruning still applies, since
+ * parquet builds those ranges and hands back whole any block it has no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a read that throws in between leaves them reachable for {@link #close()}.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key column.
+ List<Type> requestedFields = requestedSchema.getFields();
+ boolean anyNonKeyPresent = false;
+ for (int i = 0; i < requestedFields.size() && !anyNonKeyPresent; i++) {
+ anyNonKeyPresent = !keyTopLevelNames.contains(requestedFields.get(i).getName())
+ && !missingColumns.contains(parquetColumn.children().apply(i));
+ }
+ if (!anyNonKeyPresent) return;
+ nonKeyColumns = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped, with every projected byte of the rows the pushed data filter kept as
+ * avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Exception e) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. Giving the row group up hands it to phase 2 whole, which reads
+ // the same pages in the same batches as a plain scan. So the page fails at the same batch,
+ // after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain read.
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) splicedBytes += appendSurvivorRowToAccumulators(r);
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together
+ // here.
+ long rangeBytes = survivorRangeCount * ROW_RANGE_BYTES;
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ } catch (Exception e) {
+ // Fail open on any exception. The predicate runs here without the conjuncts before it in
+ // the plan, so it can throw on a row a plain scan never evaluates it on (see
+ // `ParquetStorageFilter.isSupportedStorageFilter`). Giving the filter up returns every
+ // row the pushed filter kept, and the post-scan Filter decides them in its own order.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK error
+ // (`timestamp_seconds` of a decimal), and a Hive UDF throws a checked SparkException.
+ // Rethrowing would fail a query a plain read answers, or drop the rest of the file
+ // silently under `ignoreCorruptFiles`. Only a task kill or a fatal error is rethrown,
+ // first. A failed copy of a surviving key lands here too, and gets the same answer.
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, ROW_GROUP_GIVEN_UP_ON_EVALUATION);
+ return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, and returns what that costs the
+ * budget up front. A variable-length key's byte child is sized to what the set queued before it
+ * held, so it does not grow from the default one doubling at a time for every capacity's worth
+ * of survivors. That memory is taken before any value arrives, so it is charged now, as if those
+ * values had arrived, and the values written into it use that charge up before they add to it.
+ */
+ private long allocateKeyAccumulators() {
+ WritableColumnVector[] previous = survivorBatches.peekLast();
+ // Assigned before the loop, so an allocation failure part way through leaves the vectors
+ // allocated so far reachable for `abandonSplicing`.
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ long charged = 0L;
+ for (int i = 0; i < keyColumns.length; i++) {
+ WritableColumnVector accumulator = newKeyVector(i);
+ currentKeyAccumulators[i] = accumulator;
+ prepaidValueBytes[i] = 0L;
+ if (keyColumns[i].variableLength() && previous != null) {
+ int valueBytes = previous[i].arrayData().getElementsAppended();
+ accumulator.arrayData().reserve(valueBytes);
+ prepaidValueBytes[i] = VARIABLE_LENGTH_BYTES_FACTOR * valueBytes;
+ charged += prepaidValueBytes[i];
+ }
+ }
+ return charged;
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, allocating them first if
+ * there are none, and hands them to the queue once they are full. Returns the bytes the row
+ * added, which the caller weighs against its budget.
+ */
+ private long appendSurvivorRowToAccumulators(int srcRow) {
+ long charged = currentKeyAccumulators == null ? allocateKeyAccumulators() : 0L;
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final KeyColumn[] keys = keyColumns;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ KeyColumn key = keys[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ key.copier().copy(dst, dstRow, src, srcRow);
+ // Measured on the destination, because a dictionary-encoded source has no length of its
+ // own, its values being read through the dictionary.
+ if (key.variableLength()) {
+ long valueCharge = VARIABLE_LENGTH_BYTES_FACTOR * dst.getArrayLength(dstRow);
+ long prepaid = Math.min(prepaidValueBytes[i], valueCharge);
+ prepaidValueBytes[i] -= prepaid;
+ charged += valueCharge - prepaid;
+ }
+ }
+ }
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) pushAccumulators();
+ return keyFixedBytesPerRow + charged;
+ }
+
+ /** Hands the accumulators to the queue the emit path takes its batches from. */
+ private void pushAccumulators() {
+ survivorBatches.addLast(currentKeyAccumulators);
+ currentKeyAccumulators = null;
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /** Releases the pages phase 2 read for the row group just emitted, if any. */
+ private void closeDataPages() {
+ if (dataPages != null) {
+ dataPages.close();
+ dataPages = null;
+ }
+ }
+
+ /** An error applying the filter to a row gave one row group's filter up. */
+ private static final String ROW_GROUP_GIVEN_UP_ON_EVALUATION =
+ "Reading a row group of {} without the storage filter, because applying it to a row raised "
+ + "an error. The filter is still applied above the scan, so the answer is unchanged, and "
+ + "the remaining row groups are filtered as usual";
+
+ /** An error decoding a key page gave one row group's filter up. */
+ private static final String ROW_GROUP_GIVEN_UP_ON_DECODING =
+ "Reading a row group of {} without the storage filter, because decoding its key columns "
+ + "raised an error. The row group is read the way a plain scan reads it, which raises "
+ + "the same error if the data is corrupt, and the remaining row groups are filtered as "
+ + "usual";
+
+ /** An error applying the filter to a file's constant key values gave the whole file up. */
+ private static final String FILE_GIVEN_UP_ON_EVALUATION =
+ "Reading {} without the storage filter, because applying it to the values its missing key "
+ + "columns read as raised an error. The filter is still applied above the scan, so the "
+ + "answer is unchanged";
+
+ /**
+ * Reports the first time this reader gives the filter up on an error, in one of the messages
+ * above. Once per reader, which is once per split, because a file whose values do that tends to
+ * do it again.
+ */
+ private void logFilterGivenUpOnError(Exception e, String message) {
Review Comment:
Done. The three give-up catches call `giveUpOnError`, which rethrows first and then reports. The allocation catch from your first item is the one that does not, on purpose, and its comment says why.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -151,6 +151,66 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
}
}
+ /**
+ * The conjuncts of `afterScanFilters` the file format can evaluate at the storage layer for late
+ * materialization, to prune the IO of the columns the filter does not reference.
+ *
+ * They stay in the post-scan `Filter` as well, the way a pushed data filter does. The reader is
+ * offered them, not obliged to honor them, so the plan keeps the exact check. What it costs is
+ * evaluating the conjunct a second time for the rows the scan emits. What it buys is a reader
+ * free to give up on a file it cannot read in part, or on a stretch of rows whose survivors cost
+ * too much to keep track of, without the answer depending on it.
+ *
+ * Two of the conditions are per scan, and failing either offers nothing:
+ * - [[FileFormat.supportsStorageFilterPushdown]] holds. That is where a format reads the conf
+ * that enables this, so a format's own conf never decides for another format, and asking it
+ * first keeps everything below off the path of a scan that will not use it.
+ * - [[FileFormat.supportBatch]] holds for the schema the reader will see,
+ * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives
+ * its own vectorized-read decision from. Late materialization needs a batch read, so this asks
+ * about batch support rather than naming a format.
+ *
+ * The rest are per conjunct:
+ * - It is deterministic. The reader drops the rows the conjunct rejects and the post-scan
+ * `Filter` evaluates it again on the rows it keeps, so the two evaluations have to agree.
+ * - It references at least one column, and every column it references is a projected data
+ * column. A reference to something the scan does not read cannot be evaluated by the reader.
+ * - [[FileFormat.supportsStorageFilter]] accepts it. That is where the expression shapes and
+ * column types a reader can evaluate live, so this method names neither a format nor a type.
+ *
+ * One last condition is on the set that survives, that at least one projected column is left for
+ * the reader to prune. A scan that projects nothing but the filter's own storage-key columns
+ * reads the same columns for the same rows either way, since the reader has to read a key column
+ * to evaluate the filter on it, so offering it could only add the cost of evaluating the
+ * predicate outside the generated code. Nothing is offered in that case.
+ */
+ private def storageFiltersFor(
+ afterScanFilters: ExpressionSet,
+ fsRelation: HadoopFsRelation,
+ readDataColumns: Seq[Attribute],
+ outputDataSchema: StructType): Seq[Expression] = {
+ val sparkSession = fsRelation.sparkSession
+ if (!fsRelation.fileFormat.supportsStorageFilterPushdown(sparkSession)) return Nil
+ val resultSchema = StructType(fsRelation.partitionSchema.fields ++ outputDataSchema.fields)
+ if (!fsRelation.fileFormat.supportBatch(sparkSession, resultSchema)) return Nil
+
+ val dataAttrs = AttributeSet(readDataColumns)
+ // Over `toSeq` rather than the set, because `ExpressionSet.filter` hands the predicate
+ // `e.canonicalized`, which drops attribute names and metadata, and a format deciding by either
+ // would answer about an expression it will never be given. Normalized to the relation's own
+ // column names, as the pushed data filters are, because the analyzer spells a reference the way
+ // the query did, while the reader sees the names of the schema it reads.
+ val offered = DataSourceStrategy.normalizeExprs(afterScanFilters.toSeq, readDataColumns)
Review Comment:
Done. `storageFiltersFor` takes `ExpressionSet(normalizedFilters).toSeq`, with no second normalization and no `deterministic` re-check. Its scaladoc says where the names come from.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2757 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkException, TaskContext}
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading the
+ // rest of a file without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender("partial-read warning")
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val warned = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering"))
+ (result.get, warned)
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString),
+ useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(195L))
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs the reader's own initialization, and it
+ // used to run that a second time on top. A reader that declined the filter on the first pass,
+ // here because every projected column is a key column, then reached the filter it had just
+ // released and failed with an NPE instead of reading plainly.
+ withTempDir { dir =>
+ val keys = (1L to 20L)
+ val path = writeKeyParquetFile(dir, keys)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(15L))),
+ requested)
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ Utils.tryWithResource(new LateMaterializationParquetRecordReader(false, 4096, filter)) {
+ reader =>
+ reader.initialize(split, context)
+ reader.initBatch(new StructType(), null)
+ val read = mutable.ArrayBuffer[Long]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ assert(read == keys, s"every row must come back unfiltered; got ${read.size}")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(0L))),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = ThrowsCheckedOn(BoundReference(0, LongType, nullable = false), bad = 7L)
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises a checked SparkException, so the kill is
+ // read off the task context rather than off the error. A fatal error a UDF wrapped is found in
+ // the cause chain, which is where the executor looks for one. Both are rethrown as they are.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def readWith(key: Expression): Unit = {
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ readAll(path, filter)._2.close()
+ }
+
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.setTaskContext(context)
+ val killed = try {
+ intercept[SparkException](readWith(ThrowsCheckedOn(bound, bad = 7L)))
+ } finally {
+ TaskContext.unset()
+ }
+ assert(killed.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task; got $killed")
+
+ val fatal = intercept[SparkException](
+ readWith(ThrowsCheckedOn(bound, bad = 7L, fatalCause = true)))
+ assert(fatal.getCause.isInstanceOf[OutOfMemoryError],
+ s"the error wrapping a fatal one must reach the task as it is; got $fatal")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sortBy(_._1)}; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. Phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
+ val pushed = scanOf(df.queryExecution.executedPlan).simpleString(200)
+ assert(pushed.contains("GreaterThan(v,"),
+ s"the data filter must actually be pushed for this test to mean anything: $pushed")
+ // Storage filter on `k` (key column).
+ val withSF = scanWithStorageFilter(df, threshold = 100L)
+
+ val collected = executePlanCollect(withSF).toSet
+ // Every row satisfies v > 'v_000', so the storage filter alone decides the result.
+ val expected = rows.filter(_._1 >= 100L).toSet
+ assert(collected == expected,
+ s"got ${collected.toSeq.sortBy(_._1)}; expected ${expected.toSeq.sortBy(_._1)}")
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSet == Set((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty)
+ .getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSet == Set(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again, same rows, more bytes") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Two things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. And
+ // the fallback must actually have happened, which is observable. It reads the key column a
+ // second time, so it transfers strictly more bytes. Without that second assertion the test
+ // would pass even if the cap never reached the reader.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up past the cap still
+ // has to emit in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = kvSchema()
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") {
+ readCounting(path, formatReader(fileSchema, Nil, storageFilters, countingHadoopConf()))
+ }
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the
+ // row ranges. `k >= 350` keeps a contiguous run, so there is one range to hold.
+ val (plainRows, plainBytes) = run("100b")
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity = 16)
+ try {
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected a row group with no survivor at all; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group past the cap charges its second key read against the byte metric") {
+ // Giving splicing up means phase 2 reads the key columns a second time, while the baseline
+ // counts them once, in phase 1. That extra read is a cost against the saving, so the same file
+ // and filter must report less avoided than they do while splicing. Without that term the metric
+ // would credit the feature with bytes it did transfer.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = kvSchema()
+
+ def avoidedBytes(cap: Long, threshold: Long): Long = {
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))),
+ fileSchema, m, cap)
+ val (_, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => b.column(0).getLong(i), capacity = 16)
+ try m.bytesAvoidedByPageFiltering.value finally reader.close()
+ }
+
+ val spliced = avoidedBytes(64L * 1024 * 1024, 350L)
+ val plain = avoidedBytes(100L, 350L)
+ assert(spliced > 0, s"page filtering has to avoid something here; got $spliced")
+ assert(plain < spliced,
+ s"the second key read must count against the saving; plain=$plain spliced=$spliced")
+
+ // The two halves of the budget are weighed together, and this is the case that says so. Eight
+ // survivors of a long key buffer 8 * (1 + 8) = 72 bytes and fall in one range of 40, so
+ // neither half reaches the 100-byte cap on its own while the sum passes it at the seventh.
+ // Weighing them separately would keep splicing here, and report the larger saving for it.
+ val eightSpliced = avoidedBytes(64L * 1024 * 1024, 393L)
+ val eightPlain = avoidedBytes(100L, 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+ }
+ }
+
+ test("an accumulator sized up front is charged for its byte child when it is sized") {
+ // Every accumulator of a row group after the first sizes a variable-length key's byte child to
+ // what the one before it held, before any value arrives. Charged only as values arrive, the
+ // last accumulator of a row group would hold a whole accumulator's worth of memory for a few
+ // survivors' charge. Six survivors of a 1000-byte key over a capacity of 4 are charged about
+ // 24KB value by value, and about 32KB once the second accumulator's byte child is charged when
+ // it is sized. A cap between the two gives splicing up only in the second case, which the
+ // second read of the key column shows.
+ withTempDir { dir =>
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), REPEAT('x', 996)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val fileSchema = kvSchema(keyType = StringType)
+ val storageFilters = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "4") {
+ readCounting(path, formatReader(fileSchema, Nil, storageFilters, countingHadoopConf()))
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ val (cappedRows, cappedBytes) = run("28000b")
+ assert(splicedRows == 6 && cappedRows == 6,
+ s"the filter keeps keys 15..20 either way; got $splicedRows and $cappedRows")
+ assert(cappedBytes > splicedBytes,
+ "the second accumulator's byte child must count against the cap, so the row group gives " +
+ s"splicing up and reads the key column again; capped=$cappedBytes spliced=$splicedBytes")
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index does not cost the file its page filtering") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, and the
+ // reader answers that by reading no part of a row group for the rest of the file. The pushed
+ // data filter's page pruning survives that, so the rows would not show it. The WARN the
+ // exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(50L)))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(Array.fill(length)(0xFF.toByte))
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. Giving the row group up hands it to phase 2 whole, which reads the
+ // same pages in the same batches as a plain read, so the page fails at the same batch, after
+ // the same rows. That is what makes `ignoreCorruptFiles` keep exactly what it keeps of a plain
+ // read, and it is asserted here on the readers themselves, as the rows each hands back before
+ // it throws. The filter is selective, so a reader that kept applying it would return fewer.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ val outDir = new File(dir, "damaged").getAbsolutePath
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ rows.toDF("k", "v").repartition(1).write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 64L * 1024 * 1024)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .option(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT, 100L)
+ .parquet(outDir)
+ val path = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ .head.getAbsolutePath
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(false, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(false, 16, storageFilter)
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ (read.toSeq, error)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError) = readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry leaves the reader unable to narrow the rest of this file, since the next row
+ // group's store may be as unreadable as this one's. What it must not give up is the pushed data
+ // filter's own page pruning, which parquet did before the reader saw the block, and which a
+ // plain read of the same query keeps. Dropping that would read more than having no storage
+ // filter at all.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warned) = withPartialReadWarning(read(evenKeys))
+ assert(warned, "the read must have discovered the damaged index, or this test proves nothing")
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry has latched.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the splice queues and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = kvSchema()
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString), capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // Each survivor costs 9 bytes of long key and 40 of its own range. The sum passes the cap of
+ // 1024 at survivor 21, which releases the buffer, and the ranges alone at survivor 26, which
+ // gives the filter up. The capacity decides what the buffer holds when it is released. At
+ // 4096 that is one partly filled set of accumulators, and at 16 a full set in the queue as
+ // well. Both have to end with the queue empty and the filter given up. Keeping one without
+ // the other would leave phase 2 reading non-key columns while emit spliced keys from a queue
+ // holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString))
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSet == Set("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does, since the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers conjuncts under the relation's own column names") {
+ // The analyzer spells a reference the way the query did, so under case-insensitive resolution
+ // a bloom can name `_TMP_METADATA_ROW_INDEX` while the relation, and the schema the reader
+ // requests, call the column `_tmp_metadata_row_index`. The planner normalizes what it offers to
+ // the relation's names, so the format's gate sees the name the reader will see and declines.
+ // Without that the gate passed it, `ParquetStorageFilter.create` rejected it on the driver, and
+ // a query that runs with the conf off failed with it on.
+ withTempDir { dir =>
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", s"id AS $rowIndexName", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", rowIndexName, "v")
+ .queryExecution.optimizedPlan
+ def column(name: String): Attribute =
+ relation.output.find(_.name == name).getOrElse(fail(s"no $name in the relation output"))
+
+ def offered(key: Attribute): Seq[Expression] = {
+ val logical = LogicalFilter(
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ scanOf(physical).storageFilters
+ }
+
+ // As a control, an ordinary key spelled in another case is offered, under its own name.
+ val upperK = offered(column("k").withName("K"))
+ assert(upperK.size == 1, s"a bloom over `K` must reach the scan; got $upperK")
+ assert(upperK.head.references.map(_.name).toSet == Set("k"),
+ s"and it must carry the relation's name for the column; got ${upperK.head}")
+
+ val upperRowIndex =
+ offered(column(rowIndexName).withName(rowIndexName.toUpperCase(Locale.ROOT)))
+ assert(upperRowIndex.isEmpty,
+ s"a bloom over the row-index column must not be offered in any case; got $upperRowIndex")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for. A dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`, since `/` is floating-point division in Spark SQL, which would hand every
+ // key expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize, dictionary = dictionary)
+ }
+
+ // Every column chunk of the file, from its footer. The facts the tests below assert about a
+ // fixture are read off these rather than assumed from the write options.
+ private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] =
+ footerOf(filePath).getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq
+
+ private def footerOf(filePath: String): ParquetMetadata = {
+ val conf = spark.sessionState.newHadoopConf()
+ ParquetFooterReader.readFooter(
+ HadoopInputFile.fromPath(new Path(filePath), conf), ParquetMetadataConverter.NO_FILTER)
+ }
+
+ // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String, column: String = "k"): Set[Encoding] =
+ footerChunks(filePath).filter(_.getPath.toDotString == column)
+ .flatMap(_.getEncodings.asScala).toSet
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // builds the plain vectorized reader rather than the late-materialization one. Every read helper
+ // in this suite goes through here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need. The caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ if (storageFilter == null) {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } else {
+ new LateMaterializationParquetRecordReader(useOffHeap, capacity, storageFilter)
+ }
+ } { reader =>
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // A key value as a string in its *internal* representation, so the splicing path and the plain
+ // path can be compared without knowing the type and without external type conversion.
+ // `InternalRow.getAccessor` does the per-type read, which is why this takes the batch's row
+ // rather than the column vector.
+ private def renderValue(row: InternalRow, ordinal: Int, dt: DataType): String =
+ InternalRow.getAccessor(dt)(row, ordinal) match {
+ case null => "null"
+ // Two equal arrays are two objects, so the default toString would compare identities.
+ case bytes: Array[Byte] => bytes.mkString(",")
+ case value => value.toString
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path. Whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.getRow(i), 0, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage, one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why. There is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ } {
+ val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded"
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // This is a regression test. TimeType is an AtomicType that passes every planning-time gate
+ // (it is batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a
+ // TIME join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // What FileSourceStrategy offers, what ParquetStorageFilter.create admits and what the reader
+ // can copy are one list, because isSupportedKeyType answers from ValueCopier. Both are asserted
+ // here. A type the copier has no case for would be a task failure rather than a planning-time
+ // rejection, and the delegation is what rules that out.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ assert(ValueCopier.forType(dt) != null, s"$dt should have a value copier")
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ intercept[IllegalStateException](ValueCopier.forType(dt))
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, which fixes the key row's
+ // layout. The reader pairs each key column's batch slot with its key-row position, so nothing
+ // depends on that order. Every other fixture mentions its keys in column order, and production
+ // does not promise that, since the conjuncts arrive in `afterScanFilters` order, which follows
+ // the join. So this has two key columns mentioned the other way round, and asserts both the
+ // sorted layout and that every key still comes back with its own row.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // The emit path rewrites the key slots alone, and a partition slot is not one of them. It sits
+ // past the data columns and must keep the constant vector `initBatch` populated for it.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite. OffHeapColumnVector.close() frees
+ // the native buffer, so a read after close is a crash rather than stale data. A second close
+ // is harmless, since close() zeroes the addresses it freed. capacity = 16 over 71 survivors
+ // forces 5 emits, each closing the previous emit's dequeued key vectors. The other 29 rows
+ // are what makes a declining reader fail this.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(30L))
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (30L to 100L),
+ s"expected the 71 surviving keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // One ColumnarBatch is handed out for the whole read and its key slots are rewritten per
+ // batch, so a consumer holding on to an earlier getCurrentValue() would read the wrong
+ // vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new LateMaterializationParquetRecordReader(false, 8, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution, a column missing from the physical file -----
+
+ test("non-key column missing from a file: the evolved table reads and credits avoided bytes") {
+ // Schema evolution end to end, with the non-key column rather than the key missing from the
+ // older file. The clipped requested schema keeps the missing column, which the byte walks skip
+ // by themselves, since they go over the block's chunks and the older file has no chunk for it.
+ // What this pins is that such a read returns the right rows and still credits a saving.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // The older file has (k, v), the newer one (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSet
+ val expected = ((150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))).toSet
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // A positive total says the byte walks ran over the older file's row groups as well.
+ // Asserting non-negativity would prove nothing. `SQLMetric.add` drops a negative and the
+ // value cannot go below zero.
+ val bytesRg = withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value + bytesPf.value > 0,
+ s"the walk must credit some avoided bytes; got rg=${bytesRg.value} pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = scanOf(spark.read.parquet(path).select("k", "v").queryExecution.executedPlan)
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Set[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Set[(Long, Option[Long])]) = {
+ val withSF = withStorageFilters(spark.sql(s"SELECT id, k FROM $table")) { attr =>
+ Seq(buildFilter(attr("k")))
+ }
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSet
+ (withSF, collected)
+ }
+
+ test("all keys missing: an error on the constant reads the file rather than skipping it") {
+ // A file with none of the filter's key columns makes the predicate constant over it, so the
+ // reader decides once, from the values `initBatch` materialized for those columns. That
+ // evaluation can throw exactly as a row's value can. Here the older file's `k` reads as its
+ // DEFAULT, which is not a number, and the cast fails under ANSI. Failing open reads the file
+ // the way a plain scan would, so its rows survive. Skipping it would lose them, and rethrowing
+ // would fail a query a plain read answers.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true",
+ SQLConf.ANSI_ENABLED.key -> "true") {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql("ALTER TABLE evolved ADD COLUMN k STRING DEFAULT 'not-a-number'")
+ spark.sql("INSERT INTO evolved VALUES (4, '40'), (5, '50')")
+ val withSF = withStorageFilters(spark.sql("SELECT id, k FROM evolved")) { attr =>
+ Seq(GreaterThanOrEqual(Cast(attr("k"), LongType), Literal(1L)))
+ }
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val ids = collectRows(withSF).map(_.getLong(idPos)).toSet
+ assert(ids == Set(1L, 2L, 3L, 4L, 5L),
+ s"the older file must fail open and the newer one pass the cast; got $ids")
+ }
+ }
+ }
+
+ test("all keys missing: a checked exception on the constant reads the file") {
+ // The constant evaluation fails open under the same rule as a row's, a checked exception
+ // included. The older file's `k` reads as its DEFAULT 7, which the key expression throws on,
+ // so that file must be read the way a plain scan would. The newer file's keys evaluate
+ // normally and are rejected, since no Filter runs above this scan.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(ThrowsCheckedOn(k, bad = 7L), Literal(100L)))
+ assert(collected == Set((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L))),
+ s"the older file must fail open and the newer one be filtered; got $collected")
+ }
+ }
+ }
+
+ test("a file with none of the projected non-key columns is read plainly") {
+ // The clipped requested schema keeps a column the file does not have, so a projected non-key
+ // column added after the older file was written still looks like something for phase 2 to
+ // read. Phase 2 reads nothing for it, so for that file the filter could only add cost, and the
+ // reader declines it there. With no Filter above this scan, the older file's rows come back
+ // unfiltered while the newer file's are still filtered.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTable("evolved_value") {
+ spark.sql("CREATE TABLE evolved_value (k BIGINT, v STRING) USING parquet")
+ spark.sql("INSERT INTO evolved_value VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+ spark.sql("ALTER TABLE evolved_value ADD COLUMN w STRING")
+ spark.sql("INSERT INTO evolved_value VALUES (4, 'd', 'x'), (5, 'e', 'y')")
+ val withSF =
+ scanWithStorageFilter(spark.sql("SELECT k, w FROM evolved_value"), threshold = 5L)
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val keys = collectRows(withSF).map(_.getLong(kPos)).toSet
+ assert(keys == Set(1L, 2L, 3L, 5L),
+ s"the older file must be read unfiltered and the newer one filtered; got $keys")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test. The default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Set[(Long, Option[Long])] = Set((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(StorageFilterMetrics.ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Set((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got ${rejectsNull.toSeq.sorted}")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got ${acceptsNull.toSeq.sorted}")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ val collected = collectRows(withSF).map(r => (r.getLong(0), r.getLong(2))).toSet
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1)).toSet
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got " +
+ s"${collected.toSeq.sorted.take(5)}")
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF =
+ scanWithStorageFilter(spark.read.parquet(outDir).select("k", "s"), threshold = 195L)
+ val collected = collectRows(withSF)
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSet
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k")).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sorted}; expected $expected")
+ }
+ }
+ }
+
+ test("a storage filter binds to the right column with partition and metadata columns in output") {
+ // `preparedStorageFilters` binds the filter's references against `output.take(requiredSchema
+ // .length)`, and `output` is the data columns, then the generated metadata columns, then the
+ // partition columns, then the constant metadata ones. A projection carrying all four is where a
+ // wrong prefix would bind the predicate to another column and drop rows the filter keeps, which
+ // the post-scan Filter cannot put back. The file's first column `a` is not projected, so the
+ // key sits at a different position in the relation's data schema than in the scan's.
+ withTempDir { dir =>
+ val outDir = new File(dir, "parted").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id * 7 AS a", "id AS k", "CONCAT('v_', id) AS v", "CAST(id % 3 AS INT) AS p")
+ .repartition(1)
+ .write
+ .partitionBy("p")
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select(
+ col("k"), col("v"), col("p"),
+ col("_metadata.row_index").as("ri"), col("_metadata.file_size").as("fs"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ // The assumption the binding rests on, stated where a change to `output` would break it.
+ assert(withSF.output.take(withSF.requiredSchema.length).map(_.name) ==
+ withSF.requiredSchema.fieldNames.toSeq,
+ s"the scan's leading attributes must be its data columns; got " +
+ s"${withSF.output.map(_.name)}")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val pPos = withSF.output.indexWhere(_.name == "p")
+ def collectAll(plan: SparkPlan): Set[Seq[Any]] =
+ collectRows(plan).map(_.toSeq(plan.output.map(_.dataType))).toSet
+ val filtered = collectAll(withSF)
+ // The same scan without the filter is the oracle, so the row indexes and file sizes come
+ // from the read rather than from arithmetic over how the partitions were written.
+ val expected = collectAll(withSF.copy(storageFilters = Nil))
+ .filter(_(kPos).asInstanceOf[Long] >= 150L)
+ assert(filtered == expected,
+ s"expected ${expected.size} rows with their own partition and metadata values; " +
+ s"got ${filtered.size}")
+ assert(filtered.map(_(pPos)).size == 3,
+ s"every partition value must still be represented; got ${filtered.map(_(pPos))}")
+ }
+ }
+ }
+
+ // ----- Planner gates -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false. It may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ //
+ // This hook answers about shapes and types alone. Whether the conjunct is deterministic and
+ // whether its references are projected columns of this scan are plan-visible, so they stay with
+ // the planner; `FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter`
+ // covers that half.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ // A cast key is supported. In ANSI mode it can throw on a row an earlier conjunct would have
+ // dropped, and the reader handles that where it arises, by giving the filter up for the row
+ // group. Declining here instead would also decline every widening cast, which is what type
+ // coercion inserts for a join between an int and a bigint column.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+
+ // The conf is read in the format rather than in the planner, and it is the per-scan question
+ // that carries it, asked once before anything per conjunct.
+ assert(!format.supportsStorageFilterPushdown(spark), "the feature is off with the conf off")
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ assert(format.supportsStorageFilterPushdown(spark), "and on with it on")
+ assert(!subclass.supportsStorageFilterPushdown(spark), "a subclass must not claim support")
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ assert(format.supportsStorageFilter(onCast),
+ "a cast key is pushed, and an evaluation error gives the row group up")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The rows are right either way, since the bloom stays in the post-scan Filter. What this gate
+ // saves is offering the scan a filter its reader cannot apply.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one. Every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSet
+ assert(keys == (1L to 50L).toSet,
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support declines rather than delegating") {
+ // The default `FileFormat.buildReaderWithStorageFilters` answers None, and the caller falls
+ // back to the ordinary builder. That is what keeps the two builders from being able to call
+ // each other, and declining is safe because the post-scan Filter still holds the conjunct.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(1L)))
+ val declined = new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration(), Map.empty)
+ assert(declined.isEmpty, "the default must not build a reader of its own")
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilterPushdown(spark),
+ "and it must not claim support either")
+ }
+
+ test("a scan whose format declines falls back to the ordinary reader and returns every row") {
+ // The check above is that the default answers None. This is the other half, that `inputRDD`
+ // then builds the ordinary reader and the scan still produces its rows. A `ParquetFileFormat`
+ // subclass is the shape that reaches it, since it inherits the override and declines by class,
+ // and its ordinary builder is the one an extension would have customized.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, threshold = 195L)
+ val declining = withSF.copy(
+ relation = withSF.relation.copy(fileFormat = new ParquetFileFormat() {})(spark))
+ val collected = executePlanCollect(declining).toSet
+ assert(collected == rows.toSet,
+ s"a declining format reads the file plainly; got ${collected.size} of ${rows.size} rows")
+ }
+ }
+ }
+
+ Seq(false, true).foreach { aqe =>
+ test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") {
+ // AQE is on by default in production, and it is where the bloom subquery is planned by
+ // PlanAdaptiveSubqueries rather than PlanSubqueries, which is the path
+ // preparedStorageFilters' ScalarSubquery materialization depends on.
+ withBloomFilterTables {
+ val baseConf = Map(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString)
+ def run(pushdown: Boolean): (Int, Set[(Long, Long)]) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, rows) = runBloomFilterJoin()
+ (countBloomFiltersInStorageFilters(plan),
+ rows.map(r => (r.getLong(0), r.getLong(1))).toSet)
+ }
+ val (offFilters, off) = run(false)
+ val (onFilters, on) = run(true)
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ // Equal results are guaranteed by the post-scan Filter whether or not the scan applied the
+ // filter, so they alone would not notice AQE dropping it on the way to the final plan.
+ assert(offFilters == 0 && onFilters == 1,
+ s"the scan must carry the filter with the conf on and not with it off; " +
+ s"got on=$onFilters off=$offFilters")
+ }
+ }
+ }
+
+ test("canonicalization keeps a storage-filter scan distinct from a plain one") {
+ // storageFilters is in doCanonicalize and in the case class equality, which is what stops
+ // exchange and subquery reuse from serving one scan's result to the other. Reuse compares
+ // canonicalized plans, so a scan that filters must not match a plain scan of the same file, and
+ // two scans carrying the same filter must still match.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, threshold = 25L)
+ val plain = withSF.copy(storageFilters = Nil)
+ assert(withSF != plain, "case class equality must take storageFilters into account")
+ assert(!withSF.sameResult(plain),
+ s"a filtering scan must not be reusable as a plain one:\n${withSF.canonicalized}\n" +
+ s"${plain.canonicalized}")
+ // A second, independently planned scan of the same file with the same filter still
+ // matches, so reuse is not disabled wholesale. Its key attribute carries a different
+ // exprId, which is what canonicalization normalizes away.
+ val sameSF = scanWithStorageFilter(path, threshold = 25L)
+ assert(withSF.sameResult(sameSF),
+ s"two scans with the same storage filter must stay reusable:\n" +
+ s"${withSF.canonicalized}\n${sameSF.canonicalized}")
+ val otherSF = scanWithStorageFilter(path, threshold = 30L)
+ assert(!withSF.sameResult(otherSF), "a different threshold is a different result")
+ }
+ }
+ }
+
+ test("whole-stage codegen off: the row-at-a-time path still applies the extracted bloom") {
+ // The planner's extraction gate is `fileFormat.supportBatch`, which does not look at
+ // whole-stage codegen, while `FileSourceScanExec.supportsColumnar` does. So with codegen off
+ // the bloom is still extracted, `returningBatch` is false, and the reader serves the spliced
+ // batch one row at a time. That is the only combination where the planner's gate is weaker
+ // than the runtime's.
+ withBloomFilterTables {
+ val baseConf = Map(
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "false")
+ def run(pushdown: Boolean): (Set[(Long, Long)], Int, Boolean, Long) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, result) = runBloomFilterJoin()
+ val scans = plan.collect { case s: FileSourceScanExec => s }
+ val excluded = scans.filter(_.storageFilters.nonEmpty).map { s =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+ (result.map(r => (r.getLong(0), r.getLong(1))).toSet,
+ countBloomFiltersInStorageFilters(plan), scans.forall(!_.supportsColumnar), excluded)
+ }
+ val (off, _, _, _) = run(false)
+ val (on, storageBlooms, noColumnarScan, excluded) = run(true)
+ assert(storageBlooms >= 1, s"the bloom must still be extracted with codegen off; " +
+ s"got $storageBlooms")
+ assert(noColumnarScan, "with codegen off no scan should output columnar batches")
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ // The assertions above hold whether or not the row path honored the filter, since the
+ // post-scan Filter answers the query either way. The metrics are what say it did.
+ assert(excluded > 0, s"the row path must have excluded rows; the metrics say $excluded")
+ }
+ }
+
+ test("off-heap column vectors through the planner: spliced values survive the free") {
+ // Off-heap is where the vector lifecycle actually bites. The previous batch's key vectors are
+ // freed at the next nextBatch(), so a stale reference reads released native memory rather than
+ // old bytes. This drives it through the planner, where the batch also crosses
+ // ColumnarToRowExec.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.COLUMN_VECTOR_OFFHEAP_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 150L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 150L).toSet
+ assert(collected == expected,
+ s"got ${collected.size} rows; expected ${expected.size}. " +
+ s"first few: ${collected.toSeq.sortBy(_._1).take(3)}")
+ }
+ }
+ }
+
+ // ----- Page-level pushedFilterRanges (a strict subset of the row group) -----
+
+ test("pushed data filter narrows to a page subset: phase 1 stays aligned with the row indexes") {
Review Comment:
Removed. Its rationale, phase 1's alignment under a narrowed range, is now in the comment of the other test, which asserts the narrowing and the exact rows.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, TaskContext}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. None counts what it cost. A row group
+ * whose filter was given up reads its key columns twice and counts nothing, so the counters can
+ * show a saving on a scan that read more than it would have with the feature off.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums those skipped row groups' rows, counting per block the rows
+ * that survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned. A row group that gave
+ * splicing up reads its key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+
+ /** These counters keyed for the scan, which carries them without naming any of them. */
+ def toMap: Map[String, SQLMetric] = Map(
+ StorageFilterMetrics.ROW_GROUPS_SKIPPED -> rowGroupsSkipped,
+ StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP -> rowsExcludedByRowGroup,
+ StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP -> rowsExcludedWithinRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP -> bytesAvoidedByRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING -> bytesAvoidedByPageFiltering)
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /** A fresh set of counters, with the labels the SQL UI shows. */
+ def create(sparkContext: SparkContext): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped =
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ rowsExcludedByRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ rowsExcludedWithinRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ bytesAvoidedByRowGroup = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ bytesAvoidedByPageFiltering = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made: the map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer
+ // construction to first use on the executor.
+ @transient private lazy val predicate: BasePredicate = {
Review Comment:
Done. It is `@transient lazy val preparedPredicate` with that scaladoc, and the comment on the field is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4150485908
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. None counts what it cost. A row group
+ * whose filter was given up reads its key columns twice and counts nothing, so the counters can
+ * show a saving on a scan that read more than it would have with the feature off.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums those skipped row groups' rows, counting per block the rows
+ * that survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned. A row group that gave
+ * splicing up reads its key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+
+ /** These counters keyed for the scan, which carries them without naming any of them. */
+ def toMap: Map[String, SQLMetric] = Map(
+ StorageFilterMetrics.ROW_GROUPS_SKIPPED -> rowGroupsSkipped,
+ StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP -> rowsExcludedByRowGroup,
+ StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP -> rowsExcludedWithinRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP -> bytesAvoidedByRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING -> bytesAvoidedByPageFiltering)
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /** A fresh set of counters, with the labels the SQL UI shows. */
+ def create(sparkContext: SparkContext): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped =
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ rowsExcludedByRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ rowsExcludedWithinRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ bytesAvoidedByRowGroup = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ bytesAvoidedByPageFiltering = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made: the map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ require(boundExpression != null,
+ "a storage filter must only be evaluated on an executor, and this copy has released the " +
+ "expression it would build its predicate from")
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader evaluates
+ * the predicate without the conjuncts that precede it in the plan, so one that throws on a row an
+ * earlier conjunct would have rejected throws where a plain scan does not. That is handled where
+ * it arises. The reader gives the filter up for the row group and reads it plainly, and the
+ * post-scan `Filter` then evaluates every conjunct in its own order.
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while applying a storage filter that it must not absorb by
+ * giving the filter up. Two kinds are not the reader's to absorb, and both are asked the way the
+ * rest of Spark asks them:
+ * - a pending task kill, since the error may be the kill's interrupt surfacing through a UDF
+ * that wrapped it. `PythonRunner` reads a kill off the task context in the same way.
+ * - a fatal error in the cause chain, found by `Executor.isFatalError`.
+ *
+ * Either goes out wrapped, see [[mustPropagate]].
+ */
+ def rethrowIfMustPropagate(e: Throwable): Unit = {
+ if (isKilled || Executor.isFatalError(e, fatalErrorDepth - 1)) throw mustPropagate(e)
Review Comment:
**The fatal-error check counts one wrapper, but the executor sees two.** `FileScanRDD.hasNext` (`FileScanRDD.scala:150-154`) passes every reader exception to `FileDataSourceV2.attachFilePath`, which wraps any `NonFatal` one, the `SparkException` from `mustPropagate` included, in `FAILED_READ_FILE` (`FileDataSourceV2.scala:141-142`). The executor then runs `Executor.isFatalError(t, killOnFatalErrorDepth)` (`Executor.scala:1144`) on `FAILED_READ_FILE(SparkException(e))`, two levels above `e`, while this check subtracts one.
For example, with `spark.executor.killOnFatalError.depth=3` and a phase-1 key decode whose `WritableColumnVector.reserve` wraps an OOM in a `RuntimeException`:
- here, `isFatalError(e, 2)` finds the OOM and rethrows;
- the executor walks `FAILED_READ_FILE` (3), `SparkException` (2) and `RuntimeException` (1), and stops before the OOM, so it fails the task and keeps the JVM;
- a plain read throws `FAILED_READ_FILE(RuntimeException(OOM))`, which kills the executor.
At the default depth of 5, the same happens for a fatal cause three levels below `e`. For an evaluation error, the plain plan throws `e` from `FilterExec` unwrapped, so a fatal cause one level deeper still kills there, while this check absorbs it.
So the sentence at L273-274, "It still finds a fatal cause too", does not hold. Could this throw the fatal cause itself instead? It is not `NonFatal`, so `attachFilePath` rethrows it as is (`FileDataSourceV2.scala:143`), and the executor finds it at depth 0. Otherwise, the check here could subtract two levels, with the comment fixed to match.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
Review Comment:
**Under `ignoreCorruptFiles`, these internal-invariant checks turn a reader bug into a silently truncated file.** `requireRowCount` here, the survivor queue (L365), the key ordinal and key column checks (L402, L410), `ParquetReadState.forRead` (`ParquetReadState.java:111`) and `ValueCopier.forType` (`ValueCopier.java:75`) all throw `IllegalStateException`. `DataSourceUtils.shouldIgnoreCorruptFileException` (`DataSourceUtils.scala:273`) takes any `RuntimeException` for a corrupt file, so `FileScanRDD` (`FileScanRDD.scala:284-288`) logs "Skipped the rest of the content in the corrupted file", the task succeeds, and the rest of the file is missing.
A file alone cannot reach these checks, so it takes a reader bug, but that is the case they exist for. `mustPropagate` already avoids this for a kill or a fatal error with a checked `SparkException`, and `ParquetBlockAccounting` does not throw for the same reason. Could these throw `SparkException.internalError` through a Scala helper? The one at L365 matters most, since it is the only thing between a queue mismatch and wrong key values.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
Review Comment:
Minor: this latch is wider than what failed. Parquet caches one column index store per block, and only this block's store is empty. Parquet empties it for any `IOException` while reading one offset index, a transient object-store error included. Yet from here on, every row group of the split loses in-row-group narrowing, even with intact indexes, and each one that keeps a survivor reads its key columns twice. With a pushed data filter, `getFilteredRecordCount()` has already built every block's store at initialize (`SpecificParquetRecordReaderBase.java:153`), so `fileReader.getColumnIndexStore(i).getOffsetIndex(path)` answers per block with no IO. Could the latch be per block, at least in that case?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Exception e) {
+ // A corrupt page whose decoding raises an exception shows here, since parquet reads a
+ // chunk's bytes up front but decodes a page only when it is read. Giving the row group up
+ // hands it to phase 2 whole, which reads the same pages in the same batches as a plain
+ // scan. So the page fails at the same batch, after the same rows, and `ignoreCorruptFiles`
+ // keeps what it keeps of a plain read. A codec that reports corruption as an Error, as
+ // hadoop-lzo's `InternalError` does, is not caught here and fails before the first batch.
+ giveUpOnError(e, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // A survivor vector that cannot grow, which `WritableColumnVector.reserve` reports as
+ // a RuntimeException even when an OutOfMemoryError is behind it. The buffer is this
+ // reader's own and optional, so this gives splicing up rather than the filter, and
+ // the memory goes back at once. A plain read never allocates it.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ALLOCATION);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together
+ // here.
+ long rangeBytes = survivorRangeCount * ROW_RANGE_BYTES;
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ } catch (Exception e) {
+ // Fail open on any exception. The predicate runs here without the conjuncts before it in
+ // the plan, so it can throw on a row a plain scan never evaluates it on (see
+ // `ParquetStorageFilter.isSupportedStorageFilter`). Giving the filter up returns every
+ // row the pushed filter kept, and the post-scan Filter decides them in its own order.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK error
+ // (`timestamp_seconds` of a decimal), and a Hive UDF throws a checked SparkException.
+ // Rethrowing would fail a query a plain read answers, or drop the rest of the file
+ // silently under `ignoreCorruptFiles`. Only a task kill or a fatal error is rethrown.
+ giveUpOnError(e, ROW_GROUP_GIVEN_UP_ON_EVALUATION);
+ return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
Review Comment:
**With off-heap column vectors, the survivor buffer is native memory that nothing accounts for.** `spark.sql.columnVector.offheap.enabled` falls back to `spark.memory.offHeap.enabled` (`SQLConf.scala:1063-1067`), and the accumulators are then `OffHeapColumnVector`s allocated with `Platform.allocateMemory`. That memory is outside `spark.memory.offHeap.size`, outside the heap and outside any `MemoryConsumer`, so it comes out of the container memory overhead: up to the 64MB cap per running task, about 2GB on a 32-core executor, plus the up-front `reserve` of a new accumulator set (L864), which is allocated before the cap check at L776. A plain read's vectors are not accounted for either, but they are bounded by one batch. There is no spill past the overhead, only a container OOM kill.
The `MemoryConsumer` follow-up would fix this. Until then, could the `maxSplicedRowGroupBytes` doc (`SQLConf.scala:1986-1995`) say that off heap, the buffer comes out of `spark.executor.memoryOverhead`?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Exception e) {
+ // A corrupt page whose decoding raises an exception shows here, since parquet reads a
+ // chunk's bytes up front but decodes a page only when it is read. Giving the row group up
+ // hands it to phase 2 whole, which reads the same pages in the same batches as a plain
+ // scan. So the page fails at the same batch, after the same rows, and `ignoreCorruptFiles`
+ // keeps what it keeps of a plain read. A codec that reports corruption as an Error, as
+ // hadoop-lzo's `InternalError` does, is not caught here and fails before the first batch.
+ giveUpOnError(e, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
Review Comment:
**This catch takes any `RuntimeException`, not only a vector that cannot grow.** It wraps the whole `appendSurvivorRowToAccumulators`, so an NPE or a `ClassCastException` from a `ValueCopier` or type-mismatch bug is also logged as "a vector holding the surviving key values could not grow", once per split, and only turns splicing off. Phase 2 then reads the key columns again, so the rows stay correct, and the per-type tests, "key type ...: survivors round-trip through the phase 1 accumulators" (`ParquetStorageFilterSuite.scala:1607-1631`), still pass, since they compare rows only. Only the long and string cap tests (`ParquetStorageFilterSuite.scala:814`, `:931`) compare bytes read. Before this catch, the outer one gave the filter up, `readAllWith` returned the rows the filter rejects, and those tests failed.
It also does not cover the whole buffer: `newKeyVector` (L838-843) and the `new byte[count]` in `getBinary` throw a raw `OutOfMemoryError`, which passes it by.
Could this catch only what `reserve` throws, an `OutOfMemoryError` cause or its capacity limit, or could the per-type tests assert that the row group was spliced?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2823 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading the
+ // rest of a file without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender("partial-read warning")
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val warned = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering"))
+ (result.get, warned)
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString),
+ useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(195L))
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(0L))),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = ThrowsCheckedOn(BoundReference(0, LongType, nullable = false), bad = 7L)
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. A fatal error an expression wrapped is found in the
+ // cause chain, which is where the executor looks for one. Either goes out wrapped in a checked
+ // exception, since `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def readWith(key: Expression): SparkException = intercept[SparkException] {
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ readAll(path, filter)._2.close()
+ }
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException = {
+ TaskContext.setTaskContext(TaskContext.empty())
+ try readWith(key) finally TaskContext.unset()
+ }
+ assertPropagates(killedWhile(ThrowsCheckedOn(bound, bad = 7L, killsTask = true)),
+ classOf[SparkException])
+ assertPropagates(
+ killedWhile(ThrowsCheckedOn(bound, bad = 7L, checked = false, killsTask = true)),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, which is how `WritableColumnVector.reserve`
+ // reports one.
+ val fatal = readWith(ThrowsCheckedOn(bound, bad = 7L, fatalCause = true, checked = false))
+ assertPropagates(fatal, classOf[IllegalStateException])
+ assert(fatal.getCause.getCause.isInstanceOf[OutOfMemoryError],
+ s"the fatal error must still be in the chain for the executor; got ${fatal.getCause}")
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(MarksKilledOn(bound, at = 5L), Literal(50L))), kvSchema())
+ TaskContext.setTaskContext(TaskContext.empty())
+ val thrown = try {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ } finally {
+ TaskContext.unset()
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. Phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
+ val pushed = scanOf(df.queryExecution.executedPlan).simpleString(200)
+ assert(pushed.contains("GreaterThan(v,"),
+ s"the data filter must actually be pushed for this test to mean anything: $pushed")
+ // Storage filter on `k` (key column).
+ val withSF = scanWithStorageFilter(df, threshold = 100L)
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ // Every row satisfies v > 'v_000', so the storage filter alone decides the result.
+ val expected = rows.filter(_._1 >= 100L)
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty)
+ .getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again, same rows, more bytes") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Two things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. And
+ // the fallback must actually have happened, which is observable. It reads the key column a
+ // second time, so it transfers strictly more bytes. Without that second assertion the test
+ // would pass even if the cap never reached the reader.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up past the cap still
+ // has to emit in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = kvSchema()
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") {
+ readCounting(path, formatReader(fileSchema, Nil, storageFilters, countingHadoopConf()))
+ }
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the
+ // row ranges. `k >= 350` keeps a contiguous run, so there is one range to hold.
+ val (plainRows, plainBytes) = run("100b")
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity = 16)
+ try {
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected a row group with no survivor at all; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group past the cap charges its second key read against the byte metric") {
+ // Giving splicing up means phase 2 reads the key columns a second time, while the baseline
+ // counts them once, in phase 1. That extra read is a cost against the saving, so the same file
+ // and filter must report less avoided than they do while splicing. Without that term the metric
+ // would credit the feature with bytes it did transfer.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = kvSchema()
+
+ def avoidedBytes(cap: Long, threshold: Long): Long = {
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))),
+ fileSchema, m, cap)
+ val (_, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => b.column(0).getLong(i), capacity = 16)
+ try m.bytesAvoidedByPageFiltering.value finally reader.close()
+ }
+
+ val spliced = avoidedBytes(64L * 1024 * 1024, 350L)
+ val plain = avoidedBytes(100L, 350L)
+ assert(spliced > 0, s"page filtering has to avoid something here; got $spliced")
+ assert(plain < spliced,
+ s"the second key read must count against the saving; plain=$plain spliced=$spliced")
+
+ // The two halves of the budget are weighed together, and this is the case that says so. Eight
+ // survivors of a long key buffer 8 * (1 + 8) = 72 bytes and fall in one range of 40, so
+ // neither half reaches the 100-byte cap on its own while the sum passes it at the seventh.
+ // Weighing them separately would keep splicing here, and report the larger saving for it.
+ val eightSpliced = avoidedBytes(64L * 1024 * 1024, 393L)
+ val eightPlain = avoidedBytes(100L, 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+ }
+ }
+
+ test("an accumulator sized up front is charged for its byte child when it is sized") {
+ // Every accumulator of a row group after the first sizes a variable-length key's byte child to
+ // what the one before it held, before any value arrives. Charged only as values arrive, the
+ // last accumulator of a row group would hold a whole accumulator's worth of memory for a few
+ // survivors' charge. Six survivors of a 1000-byte key over a capacity of 4 are charged about
+ // 24KB value by value, and about 32KB once the second accumulator's byte child is charged when
+ // it is sized. A cap between the two gives splicing up only in the second case, which the
+ // second read of the key column shows.
+ withTempDir { dir =>
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), REPEAT('x', 996)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val fileSchema = kvSchema(keyType = StringType)
+ val storageFilters = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "4") {
+ readCounting(path, formatReader(fileSchema, Nil, storageFilters, countingHadoopConf()))
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ val (cappedRows, cappedBytes) = run("28000b")
+ assert(splicedRows == 6 && cappedRows == 6,
+ s"the filter keeps keys 15..20 either way; got $splicedRows and $cappedRows")
+ assert(cappedBytes > splicedBytes,
+ "the second accumulator's byte child must count against the cap, so the row group gives " +
+ s"splicing up and reads the key column again; capped=$cappedBytes spliced=$splicedBytes")
+
+ // And the keys spliced from the accumulator sized up front are the right ones.
+ val (pairs, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(storageFilters, fileSchema),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString),
+ capacity = 4)
+ try {
+ assert(pairs == (15 to 20).map(id => (f"$id%04d" + "x" * 996, id.toString)),
+ s"every survivor must come back with its own key; got ${pairs.map(_._2)}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index does not cost the file its page filtering") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, and the
+ // reader answers that by no longer narrowing a row group for the rest of the split. The pushed
+ // data filter's page pruning survives that, so the rows would not show it. The WARN the
+ // exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(50L)))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(Array.fill(length)(0xFF.toByte))
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an Error instead, as hadoop-lzo does, fails before the row group's first batch.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ val outDir = new File(dir, "damaged").getAbsolutePath
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ rows.toDF("k", "v").repartition(1).write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 64L * 1024 * 1024)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .option(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT, 100L)
+ .parquet(outDir)
+ val path = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ .head.getAbsolutePath
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(false, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(false, 16, storageFilter)
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ (read.toSeq, error)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError) = readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry leaves the reader unable to narrow the rest of this file, since the next row
+ // group's store may be as unreadable as this one's. What it must not give up is the pushed data
+ // filter's own page pruning, which parquet did before the reader saw the block, and which a
+ // plain read of the same query keeps. Dropping that would read more than having no storage
+ // filter at all.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warned) = withPartialReadWarning(read(evenKeys))
+ assert(warned, "the read must have discovered the damaged index, or this test proves nothing")
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry has latched.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the survivor queue and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = kvSchema()
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString), capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // Each survivor costs 9 bytes of long key and 40 of its own range. The sum passes the cap of
+ // 1024 at survivor 21, which releases the buffer, and the ranges alone at survivor 26, which
+ // gives the filter up. The capacity decides what the buffer holds when it is released. At
+ // 4096 that is one partly filled set of accumulators, and at 16 a full set in the queue as
+ // well. Both have to end with the queue empty and the filter given up. Keeping one without
+ // the other would leave phase 2 reading non-key columns while emit spliced keys from a queue
+ // holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString))
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSeq == Seq("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does, since the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers conjuncts under the relation's own column names") {
+ // The analyzer spells a reference the way the query did, so under case-insensitive resolution
+ // a bloom can name `_TMP_METADATA_ROW_INDEX` while the relation, and the schema the reader
+ // requests, call the column `_tmp_metadata_row_index`. The planner normalizes what it offers to
+ // the relation's names, so the format's gate sees the name the reader will see and declines.
+ // Without that the gate passed it, `ParquetStorageFilter.create` rejected it on the driver, and
+ // a query that runs with the conf off failed with it on.
+ withTempDir { dir =>
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", s"id AS $rowIndexName", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", rowIndexName, "v")
+ .queryExecution.optimizedPlan
+ def column(name: String): Attribute =
+ relation.output.find(_.name == name).getOrElse(fail(s"no $name in the relation output"))
+
+ def offered(key: Attribute): Seq[Expression] = {
+ val logical = LogicalFilter(
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ scanOf(physical).storageFilters
+ }
+
+ // As a control, an ordinary key spelled in another case is offered, under its own name.
+ val upperK = offered(column("k").withName("K"))
+ assert(upperK.size == 1, s"a bloom over `K` must reach the scan; got $upperK")
+ assert(upperK.head.references.map(_.name).toSet == Set("k"),
+ s"and it must carry the relation's name for the column; got ${upperK.head}")
+
+ val upperRowIndex =
+ offered(column(rowIndexName).withName(rowIndexName.toUpperCase(Locale.ROOT)))
+ assert(upperRowIndex.isEmpty,
+ s"a bloom over the row-index column must not be offered in any case; got $upperRowIndex")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for. A dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`, since `/` is floating-point division in Spark SQL, which would hand every
+ // key expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize, dictionary = dictionary)
+ }
+
+ // Every column chunk of the file, from its footer. The facts the tests below assert about a
+ // fixture are read off these rather than assumed from the write options.
+ private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] =
+ footerOf(filePath).getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq
+
+ private def footerOf(filePath: String): ParquetMetadata = {
+ val conf = spark.sessionState.newHadoopConf()
+ ParquetFooterReader.readFooter(
+ HadoopInputFile.fromPath(new Path(filePath), conf), ParquetMetadataConverter.NO_FILTER)
+ }
+
+ // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String, column: String = "k"): Set[Encoding] =
+ footerChunks(filePath).filter(_.getPath.toDotString == column)
+ .flatMap(_.getEncodings.asScala).toSet
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // builds the plain vectorized reader rather than the late-materialization one. Every read helper
+ // in this suite goes through here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need. The caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ if (storageFilter == null) {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } else {
+ new LateMaterializationParquetRecordReader(useOffHeap, capacity, storageFilter)
+ }
+ } { reader =>
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // A key value as a string in its *internal* representation, so the splicing path and the plain
+ // path can be compared without knowing the type and without external type conversion.
+ // `InternalRow.getAccessor` does the per-type read, which is why this takes the batch's row
+ // rather than the column vector.
+ private def renderValue(row: InternalRow, ordinal: Int, dt: DataType): String =
+ InternalRow.getAccessor(dt)(row, ordinal) match {
+ case null => "null"
+ // Two equal arrays are two objects, so the default toString would compare identities.
+ case bytes: Array[Byte] => bytes.mkString(",")
+ case value => value.toString
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path. Whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.getRow(i), 0, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage, one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why. There is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ } {
+ val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded"
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // This is a regression test. TimeType is an AtomicType that passes every planning-time gate
+ // (it is batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a
+ // TIME join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // What FileSourceStrategy offers, what ParquetStorageFilter.create admits and what the reader
+ // can copy are one list, because isSupportedKeyType answers from ValueCopier. Both are asserted
+ // here. A type the copier has no case for would be a task failure rather than a planning-time
+ // rejection, and the delegation is what rules that out.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ assert(ValueCopier.forType(dt) != null, s"$dt should have a value copier")
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ intercept[IllegalStateException](ValueCopier.forType(dt))
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, which fixes the key row's
+ // layout. The reader pairs each key column's batch slot with its key-row position, so nothing
+ // depends on that order. Every other fixture mentions its keys in column order, and production
+ // does not promise that, since the conjuncts arrive in `afterScanFilters` order, which follows
+ // the join. So this has two key columns mentioned the other way round, and asserts both the
+ // sorted layout and that every key still comes back with its own row.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // The emit path rewrites the key slots alone, and a partition slot is not one of them. It sits
+ // past the data columns and must keep the constant vector `initBatch` populated for it.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite. OffHeapColumnVector.close() frees
+ // the native buffer, so a read after close is a crash rather than stale data. A second close
+ // is harmless, since close() zeroes the addresses it freed. capacity = 16 over 71 survivors
+ // forces 5 emits, each closing the previous emit's dequeued key vectors. The other 29 rows
+ // are what makes a declining reader fail this.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(30L))
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (30L to 100L),
+ s"expected the 71 surviving keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // One ColumnarBatch is handed out for the whole read and its key slots are rewritten per
+ // batch, so a consumer holding on to an earlier getCurrentValue() would read the wrong
+ // vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new LateMaterializationParquetRecordReader(false, 8, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution, a column missing from the physical file -----
+
+ test("non-key column missing from a file: the evolved table reads and credits avoided bytes") {
+ // Schema evolution end to end, with the non-key column rather than the key missing from the
+ // older file. The clipped requested schema keeps the missing column, which the byte walks skip
+ // by themselves, since they go over the block's chunks and the older file has no chunk for it.
+ // What this pins is that such a read returns the right rows and still credits a saving.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // The older file has (k, v), the newer one (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSeq.sorted
+ val expected = (150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // A positive total says the byte walks ran over the older file's row groups as well.
+ // Asserting non-negativity would prove nothing. `SQLMetric.add` drops a negative and the
+ // value cannot go below zero.
+ val bytesRg = withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value + bytesPf.value > 0,
+ s"the walk must credit some avoided bytes; got rg=${bytesRg.value} pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = scanOf(spark.read.parquet(path).select("k", "v").queryExecution.executedPlan)
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Seq[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read. The rows come
+ // back sorted, so a row returned twice shows.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Seq[(Long, Option[Long])]) = {
+ val withSF = withStorageFilters(spark.sql(s"SELECT id, k FROM $table")) { attr =>
+ Seq(buildFilter(attr("k")))
+ }
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSeq.sorted
+ (withSF, collected)
+ }
+
+ test("all keys missing: an error on the constant reads the file rather than skipping it") {
+ // A file with none of the filter's key columns makes the predicate constant over it, so the
+ // reader decides once, from the values `initBatch` materialized for those columns. That
+ // evaluation can throw exactly as a row's value can. Here the older file's `k` reads as its
+ // DEFAULT, which is not a number, and the cast fails under ANSI. Failing open reads the file
+ // the way a plain scan would, so its rows survive. Skipping it would lose them, and rethrowing
+ // would fail a query a plain read answers.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true",
+ SQLConf.ANSI_ENABLED.key -> "true") {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql("ALTER TABLE evolved ADD COLUMN k STRING DEFAULT 'not-a-number'")
+ spark.sql("INSERT INTO evolved VALUES (4, '40'), (5, '50')")
+ val withSF = withStorageFilters(spark.sql("SELECT id, k FROM evolved")) { attr =>
+ Seq(GreaterThanOrEqual(Cast(attr("k"), LongType), Literal(1L)))
+ }
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val ids = collectRows(withSF).map(_.getLong(idPos)).toSeq.sorted
+ assert(ids == Seq(1L, 2L, 3L, 4L, 5L),
+ s"the older file must fail open and the newer one pass the cast; got $ids")
+ }
+ }
+ }
+
+ test("all keys missing: a checked exception on the constant reads the file") {
+ // The constant evaluation fails open under the same rule as a row's, a checked exception
+ // included. The older file's `k` reads as its DEFAULT 7, which the key expression throws on,
+ // so that file must be read the way a plain scan would. The newer file's keys evaluate
+ // normally and are rejected, since no Filter runs above this scan.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(ThrowsCheckedOn(k, bad = 7L), Literal(100L)))
+ assert(collected == Seq((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L))),
+ s"the older file must fail open and the newer one be filtered; got $collected")
+ }
+ }
+ }
+
+ test("a file with none of the projected non-key columns is read plainly") {
+ // The clipped requested schema keeps a column the file does not have, so a projected non-key
+ // column added after the older file was written still looks like something for phase 2 to
+ // read. Phase 2 reads nothing for it, so for that file the filter could only add cost, and the
+ // reader declines it there. With no Filter above this scan, the older file's rows come back
+ // unfiltered while the newer file's are still filtered.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTable("evolved_value") {
+ spark.sql("CREATE TABLE evolved_value (k BIGINT, v STRING) USING parquet")
+ spark.sql("INSERT INTO evolved_value VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+ spark.sql("ALTER TABLE evolved_value ADD COLUMN w STRING")
+ spark.sql("INSERT INTO evolved_value VALUES (4, 'd', 'x'), (5, 'e', 'y')")
+ val withSF =
+ scanWithStorageFilter(spark.sql("SELECT k, w FROM evolved_value"), threshold = 5L)
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val keys = collectRows(withSF).map(_.getLong(kPos)).toSeq.sorted
+ assert(keys == Seq(1L, 2L, 3L, 5L),
+ s"the older file must be read unfiltered and the newer one filtered; got $keys")
+ }
+ }
+ }
+
+ test("a file with none of a projected struct's requested fields is read plainly") {
+ // Under the legacy returnNullStructIfAllFieldsMissing, the clipped schema keeps a struct none
+ // of whose requested fields the file has, and adds no field it has, so phase 2 would read
+ // nothing for it. Only the struct's leaves are missing columns, not the struct, so the reader
+ // asks the file schema about every non-key leaf. The filter rejects every row, so a declined
+ // file comes back whole and one that is not comes back empty.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 20).selectExpr("id AS k", "named_struct('a', CAST(id AS INT)) AS s"),
+ rowGroupSize = 64 * 1024L)
+ val schema =
+ new StructType().add("k", LongType).add("s", new StructType().add("b", IntegerType))
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(1000L)))
+ withSQLConf(SQLConf.LEGACY_PARQUET_RETURN_NULL_STRUCT_IF_ALL_FIELDS_MISSING.key -> "true") {
+ val keys = readLongs(formatReader(schema, Nil, storageFilters), path, ordinal = 0)
+ assert(keys == (0L until 20L), s"the file must be read plainly; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Seq[(Long, Option[Long])] =
+ Seq((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test. The default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Seq[(Long, Option[Long])] = Seq((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected, s"got $collected; expected $expected")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(StorageFilterMetrics.ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Seq((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got $rejectsNull")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Seq[(Long, Option[Long])] =
+ Seq((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got $acceptsNull")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ val collected = collectRows(withSF).map(r => (r.getLong(0), r.getLong(2))).toSeq.sorted
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1))
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got ${collected.take(5)}")
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF =
+ scanWithStorageFilter(spark.read.parquet(outDir).select("k", "s"), threshold = 195L)
+ val collected = collectRows(withSF)
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSeq.sorted
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k"))
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ test("a storage filter binds to the right column with partition and metadata columns in output") {
+ // `preparedStorageFilters` binds the filter's references against `output.take(requiredSchema
+ // .length)`, and `output` is the data columns, then the generated metadata columns, then the
+ // partition columns, then the constant metadata ones. A projection carrying all four is where a
+ // wrong prefix would bind the predicate to another column and drop rows the filter keeps, which
+ // the post-scan Filter cannot put back. The file's first column `a` is not projected, so the
+ // key sits at a different position in the relation's data schema than in the scan's.
+ withTempDir { dir =>
+ val outDir = new File(dir, "parted").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id * 7 AS a", "id AS k", "CONCAT('v_', id) AS v", "CAST(id % 3 AS INT) AS p")
+ .repartition(1)
+ .write
+ .partitionBy("p")
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select(
+ col("k"), col("v"), col("p"),
+ col("_metadata.row_index").as("ri"), col("_metadata.file_size").as("fs"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ // The assumption the binding rests on, stated where a change to `output` would break it.
+ assert(withSF.output.take(withSF.requiredSchema.length).map(_.name) ==
+ withSF.requiredSchema.fieldNames.toSeq,
+ s"the scan's leading attributes must be its data columns; got " +
+ s"${withSF.output.map(_.name)}")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val pPos = withSF.output.indexWhere(_.name == "p")
+ def collectAll(plan: SparkPlan): Seq[Seq[Any]] =
+ collectRows(plan).map(_.toSeq(plan.output.map(_.dataType))).toSeq
+ .sortBy(_(kPos).asInstanceOf[Long])
+ val filtered = collectAll(withSF)
+ // The same scan without the filter is the oracle, so the row indexes and file sizes come
+ // from the read rather than from arithmetic over how the partitions were written.
+ val expected = collectAll(withSF.copy(storageFilters = Nil))
+ .filter(_(kPos).asInstanceOf[Long] >= 150L)
+ assert(filtered == expected,
+ s"expected ${expected.size} rows with their own partition and metadata values; " +
+ s"got ${filtered.size}")
+ assert(filtered.map(_(pPos)).distinct.size == 3,
+ s"every partition value must still be represented; got ${filtered.map(_(pPos)).distinct}")
+ }
+ }
+ }
+
+ // ----- Planner gates -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false. It may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ //
+ // This hook answers about shapes and types alone. Whether the conjunct is deterministic and
+ // whether its references are projected columns of this scan are plan-visible, so they stay with
+ // the planner; `FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter`
+ // covers that half.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ // A cast key is supported. In ANSI mode it can throw on a row an earlier conjunct would have
+ // dropped, and the reader handles that where it arises, by giving the filter up for the row
+ // group. Declining here instead would also decline every widening cast, which is what type
+ // coercion inserts for a join between an int and a bigint column.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+
+ // The conf is read in the format rather than in the planner, and it is the per-scan question
+ // that carries it, asked once before anything per conjunct.
+ assert(!format.supportsStorageFilterPushdown(spark), "the feature is off with the conf off")
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ assert(format.supportsStorageFilterPushdown(spark), "and on with it on")
+ assert(!subclass.supportsStorageFilterPushdown(spark), "a subclass must not claim support")
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ assert(format.supportsStorageFilter(onCast),
+ "a cast key is pushed, and an evaluation error gives the row group up")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The rows are right either way, since the bloom stays in the post-scan Filter. What this gate
+ // saves is offering the scan a filter its reader cannot apply.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one. Every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSeq.sorted
+ assert(keys == (1L to 50L),
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support declines rather than delegating") {
+ // The default `FileFormat.buildReaderWithStorageFilters` answers None, and the caller falls
+ // back to the ordinary builder. That is what keeps the two builders from being able to call
+ // each other, and declining is safe because the post-scan Filter still holds the conjunct.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(1L)))
+ val declined = new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration(), Map.empty)
+ assert(declined.isEmpty, "the default must not build a reader of its own")
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilterPushdown(spark),
+ "and it must not claim support either")
+ }
+
+ test("a scan whose format declines falls back to the ordinary reader and returns every row") {
+ // The check above is that the default answers None. This is the other half, that `inputRDD`
+ // then builds the ordinary reader and the scan still produces its rows. A `ParquetFileFormat`
+ // subclass is the shape that reaches it, since it inherits the override and declines by class,
+ // and its ordinary builder is the one an extension would have customized.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, threshold = 195L)
+ val declining = withSF.copy(
+ relation = withSF.relation.copy(fileFormat = new ParquetFileFormat() {})(spark))
+ val collected = executePlanCollect(declining).toSeq.sorted
+ assert(collected == rows,
+ s"a declining format reads the file plainly; got ${collected.size} of ${rows.size} rows")
+ }
+ }
+ }
+
+ Seq(false, true).foreach { aqe =>
+ test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") {
+ // AQE is on by default in production, and it is where the bloom subquery is planned by
+ // PlanAdaptiveSubqueries rather than PlanSubqueries, which is the path
+ // preparedStorageFilters' ScalarSubquery materialization depends on.
+ withBloomFilterTables {
+ val baseConf = Map(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString)
+ def run(pushdown: Boolean): (Int, Seq[(Long, Long)]) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, rows) = runBloomFilterJoin()
+ (countBloomFiltersInStorageFilters(plan),
+ rows.map(r => (r.getLong(0), r.getLong(1))).toSeq.sorted)
+ }
+ val (offFilters, off) = run(false)
+ val (onFilters, on) = run(true)
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ // Equal results are guaranteed by the post-scan Filter whether or not the scan applied the
+ // filter, so they alone would not notice AQE dropping it on the way to the final plan.
+ assert(offFilters == 0 && onFilters == 1,
Review Comment:
Minor: the AQE arm checks that the final plan's scan carries the filter, but not that the reader applied it. Equal results and the filter count both hold when the reader ignores the filter, as L742-743 says. The test that checks the storage-filter metrics end to end turns AQE off (L733), as do the other end-to-end tests that set it (L763, L1350, L2237, L2362). Since AQE is the default, nothing would catch the reader falling back to a plain read only under AQE, for example `buildReaderWithStorageFilters` returning `None` there. Could this arm also assert that the scan it finds excluded some rows, as L744-750 does?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java:
##########
@@ -26,23 +27,20 @@
/**
* Helper class to store intermediate state while reading a Parquet column chunk.
+ *
+ * <p>There is one subclass per way of saying which rows to include, which is every row of the
+ * chunk, the ranges the caller of the read already held, or the one row index per row that a page
+ * store describes parquet's own filtering with. {@link #forRead} picks between them. Everything
+ * else about a read is the same whichever it is, and lives here.
*/
-final class ParquetReadState {
- /** A special row range used when there is no row indexes (hence all rows must be included) */
- private static final RowRange MAX_ROW_RANGE = new RowRange(Long.MIN_VALUE, Long.MAX_VALUE);
-
+abstract class ParquetReadState {
Review Comment:
Nit: one final class holding a `List<RowRanges.Range>` (null for every row) and a cursor could replace the three subclasses and `forRead`. Where only row indexes are given, `RowRanges.builder().addSelectedRow(...)` coalesces them into the same ranges as the `RowIndexState` loop (L221-237). `rowRangesOf` in `VectorizedRleValuesReaderSuite.scala:398-401` already relies on that, and this PR uses the builder in `LateMaterializationParquetRecordReader` (L717, L757). The constructor could then call `nextRange()` again (L121-122 explains why it cannot now), `nextRange()` would stay non-virtual on the value readers' paths, and `ParquetTestAccess` could keep reflecting the constructor.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:
##########
@@ -190,6 +191,64 @@ class ParquetFileFormat
filters: Seq[Filter],
options: Map[String, String],
hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow] = {
+ buildParquetReader(
+ sparkSession, dataSchema, partitionSchema, requiredSchema, filters, Nil, options, hadoopConf,
+ Map.empty)
+ }
+
+ /**
+ * The conf that turns this on is read here rather than in the planner, the way `supportBatch`
+ * reads its own confs, so a Parquet-named conf does not decide for a format that is not Parquet.
+ *
+ * Subclasses answer false on purpose, even though they inherit this reader, since a subclass may
+ * customize reading by overriding `buildReaderWithPartitionValues`, and a scan with storage
+ * filters routes through `buildReaderWithStorageFilters` instead, which would silently bypass
+ * whatever the subclass does.
+ */
+ override def supportsStorageFilterPushdown(sparkSession: SparkSession): Boolean =
+ getSqlConf(sparkSession).parquetStorageFilterPushdownEnabled &&
+ getClass == classOf[ParquetFileFormat]
+
+ override def supportsStorageFilter(expr: Expression): Boolean =
+ ParquetStorageFilter.isSupportedStorageFilter(expr)
+
+ /** See `StorageFilterMetrics` for what each of these counts. */
+ override def storageFilterMetrics(sparkContext: SparkContext): Map[String, SQLMetric] =
+ StorageFilterMetrics.create(sparkContext).toMap
+
+ override def buildReaderWithStorageFilters(
+ sparkSession: SparkSession,
+ dataSchema: StructType,
+ partitionSchema: StructType,
+ requiredSchema: StructType,
+ filters: Seq[Filter],
+ storageFilters: Seq[Expression],
+ options: Map[String, String],
+ hadoopConf: Configuration,
+ storageFilterMetrics: Map[String, SQLMetric])
+ : Option[PartitionedFile => Iterator[InternalRow]] = {
+ // The same subclass exclusion as `supportsStorageFilterPushdown`, asked again because this
+ // entry point is reachable without the planner having asked it, and `None` sends the caller to
+ // the ordinary builder, which is the one a subclass overrides.
+ if (getClass != classOf[ParquetFileFormat]) {
Review Comment:
Minor: this repeats the exclusion at L210, and neither copy can be overridden. A subclass that overrides only `supportsStorageFilterPushdown` gets the filter offered, `StorageFilters` in EXPLAIN and the five metrics, but this inherited check returns `None`, and the scan reads plainly with zero metrics. If only this copy were relaxed instead, a subclass that customizes `buildReaderWithPartitionValues` would be bypassed. Could both go through one private predicate, or a protected one if subclasses are meant to opt in?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Exception e) {
+ // A corrupt page whose decoding raises an exception shows here, since parquet reads a
+ // chunk's bytes up front but decodes a page only when it is read. Giving the row group up
+ // hands it to phase 2 whole, which reads the same pages in the same batches as a plain
+ // scan. So the page fails at the same batch, after the same rows, and `ignoreCorruptFiles`
+ // keeps what it keeps of a plain read. A codec that reports corruption as an Error, as
+ // hadoop-lzo's `InternalError` does, is not caught here and fails before the first batch.
+ giveUpOnError(e, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
Review Comment:
Minor: for a file without offset indexes, which pyarrow's `write_table` writes by default, this makes the feature a net loss whenever the keys are not clustered. Every row group reads its key chunks here and again in phase 2 (L545), and with unclustered keys almost every row group keeps a survivor, if only a bloom false positive. Only a row group the filter empties pays any of that back, and since nothing is reported for a row group whose filter was given up (L574), the metrics still show a saving. The doc says such a row group reads its keys twice. It does not say that for such a file, that is most row groups. Could the reader decline the filter for the rest of the file once a few row groups in a row could not be narrowed and kept a survivor, the way `partialReadUnavailable` latches?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
Review Comment:
Nit: `resultBatch()` (`VectorizedParquetRecordReader.java:378-386`) only promises one reused batch object. With this class, a key slot holds a different vector in each batch, and this line frees the previous one. A caller that takes `batch.column(i)` once before its `nextBatch()` loop, as `DataSourceReadBenchmark.scala:235` does, would then read freed memory: an NPE on heap, a crash off heap. Every in-tree consumer fetches the columns again per batch, so nothing hits this today. Since the class and its constructors are public, could its javadoc say that `column(i)` must be fetched again after every `nextBatch()`?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
Review Comment:
Minor: this comment undersells what the throw prevents. Without it, `publishedKeyVectors` stays null, and L372-373 point the key slot at `persistentBatchColumns[slot]`, which a spliced row group has reset but never read into, since `suppliesOwnVector` nulled its reader. The batch would then carry stale or uninitialized key values as non-null values, paired with this row group's rows. That is a wrong answer, not a null slot or an NPE.
Similarly, `readerRangesFor` (L654-665) presents its null as the cheaper choice, but it is required. For a whole-block request, `readFilteredRowGroup` returns a store with no row indexes, and `ParquetReadState.forRead` refuses ranges then (`ParquetReadState.java:106-114`). Could both comments state the real constraint, so that neither looks safe to simplify away?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
Review Comment:
Nit: this is the first use of `org.apache.parquet.internal.*` in the `sql` modules, which have none at the merge base. This PR imports `ColumnIndexStore.MissingOffsetIndexException` here (L36), and `ColumnIndexStore` and `OffsetIndex` in `ParquetBlockAccounting.java:30-32`, and the phase-2 backstop depends on catching that internal type. If a parquet upgrade moves it, the build breaks, which is loud. If parquet starts wrapping it, this catch silently stops matching, and a read that a plain scan answers fails here, or drops the rest of the file under `ignoreCorruptFiles`. The test at `ParquetStorageFilterSuite.scala:1204` should catch the latter on an upgrade. Could the internal API use be kept to one place, with a note to revisit it on parquet upgrades?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2823 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading the
+ // rest of a file without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender("partial-read warning")
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val warned = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering"))
+ (result.get, warned)
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString),
+ useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(195L))
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(0L))),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = ThrowsCheckedOn(BoundReference(0, LongType, nullable = false), bad = 7L)
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. A fatal error an expression wrapped is found in the
+ // cause chain, which is where the executor looks for one. Either goes out wrapped in a checked
+ // exception, since `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def readWith(key: Expression): SparkException = intercept[SparkException] {
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ readAll(path, filter)._2.close()
+ }
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException = {
+ TaskContext.setTaskContext(TaskContext.empty())
+ try readWith(key) finally TaskContext.unset()
+ }
+ assertPropagates(killedWhile(ThrowsCheckedOn(bound, bad = 7L, killsTask = true)),
+ classOf[SparkException])
+ assertPropagates(
+ killedWhile(ThrowsCheckedOn(bound, bad = 7L, checked = false, killsTask = true)),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, which is how `WritableColumnVector.reserve`
+ // reports one.
+ val fatal = readWith(ThrowsCheckedOn(bound, bad = 7L, fatalCause = true, checked = false))
+ assertPropagates(fatal, classOf[IllegalStateException])
+ assert(fatal.getCause.getCause.isInstanceOf[OutOfMemoryError],
+ s"the fatal error must still be in the chain for the executor; got ${fatal.getCause}")
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(MarksKilledOn(bound, at = 5L), Literal(50L))), kvSchema())
+ TaskContext.setTaskContext(TaskContext.empty())
+ val thrown = try {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ } finally {
+ TaskContext.unset()
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. Phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
Review Comment:
Minor: this test cannot fail for the reason its comment gives, and it does not exercise the composition either.
- With a pushed filter, `getFilteredRecordCount()` resolves and caches every block's ranges at initialize, over the full projection (`SpecificParquetRecordReaderBase.java:145`, `:153`), so phase 0 cannot see the wrong schema. Removing the defensive `setRequestedSchema(requestedColumns)` at `LateMaterializationParquetRecordReader.java:490` keeps this test green.
- Every value is above `v_000`, so the pushed ranges are the whole row group, and phase 2 reads under the storage filter alone, as L664 says.
Could the pushed filter exclude some pages of a multi-page row group, with the expected rows being those that pass both filters?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -179,13 +184,13 @@ public VectorizedParquetRecordReader(boolean useOffHeap, int capacity) {
}
/**
- * Implementation of RecordReader API.
+ * Implementation of RecordReader API. Super's delegates to the overload below, which runs
+ * {@link #initializeInternal()}, so this does not run it a second time.
*/
@Override
public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptContext)
Review Comment:
Nit: now that this no longer calls `initializeInternal()`, the override only calls `super.initialize(...)`. `SpecificParquetRecordReaderBase.initialize(InputSplit, TaskAttemptContext)` (`SpecificParquetRecordReaderBase.java:106-110`) already dispatches to the 5-arg overload at L197, which runs `initializeInternal()` once, so this override and its javadoc can go. Its JVM descriptor is the inherited one, so nothing compiled against it breaks.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java:
##########
@@ -106,6 +107,7 @@ public VectorizedColumnReader(
ColumnDescriptor descriptor,
boolean isRequired,
PageReadStore pageReadStore,
+ RowRanges rowRanges,
Review Comment:
Nit: this replaces the public 9-argument constructor instead of overloading it, so code compiled against it, such as a plugin's own Parquet reader, gets `NoSuchMethodError`. `org.apache.spark.sql.execution.*` is excluded from MiMa and this constructor has changed before, so this is not required. Still, a 3-line overload that passes `null` ranges would behave exactly like the plain path does today and keep such code working.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-6011558936
@dongjoon-hyun thank you for round 11. All 8 are done in one commit, and the description is updated.
The up-front sizing is gone, as you suggest in item 2, which also settles items 1 and 3. Every set now grows as the first one does, so what a row group is charged no longer depends on the cap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4201448448
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1032 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A fixed-width key needs no
+ * factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. It is part of `keyFixedBytesPerRow`, counted
+ * twice, for its data byte and its null byte.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // decoders of their last page.
+ closeDataPages();
+ releaseRowGroupReaders();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
Review Comment:
**[P3, optional] Consider releasing oversized scratch vectors before phase 2.**
When splicing or filtering is abandoned, phase 2 can reread keys into persistent vectors while `keyScratchVectors` retain phase 1's largest allocations. With large variable-width keys and default vector-shrinking settings, both allocations remain live. Could oversized scratch vectors be released or replaced at this boundary, using a threshold to avoid reallocating ordinary batches?
This is source-established retention, not a demonstrated OOM or violation of the documented budget.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3708 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // The native address of an off-heap vector's values, 0 once it is freed. Only for a fixed-width
+ // vector. A string or binary one keeps its bytes in a child and reports 0 even while live.
+ private def address(vector: Any): Long =
+ vector.asInstanceOf[OffHeapColumnVector].valuesNativeAddress()
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t @ (_: VirtualMachineError | _: LinkageError) =>
+ fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`, and returns the rows.
+ private def readLaterSet(path: String, cap: Long): Seq[(String, String)] = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString),
+ capacity = 4)
+ reader.close()
+ rows
+ }
+
+ test("a later accumulator set of a variable-length key is charged per value") {
+ // Six survivors over a capacity of 4 fill one set of four 1000-byte keys and a second of two
+ // 2000-byte ones. Every set grows as its values arrive, the way the first one does. Each value
+ // is charged four times its bytes plus its row's fixed part of 17, and the one range adds 40.
+ // So the row group is charged 4 * (4000 + 17) + 2 * (8000 + 17) + 40 = 32142 bytes. A cap of
+ // 32142 keeps splicing and one of 32141 gives it up, which a missing or a double charge would
+ // each get wrong. And the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ assertSpliceCharge(32142L, rows = 6) { cap =>
+ readUnderCap(path, laterSetSchema, laterSetFilter, cap, batchSize = 4)
+ }
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val rows = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ }
+ }
+ }
+
+ test("what a row group of scattered survivors is charged does not depend on the cap") {
+ // Keys 9 to 12 fill the first set over a capacity of 4, and keys 14 and 17 come scattered, each
+ // in a range of its own. Every set grows as values arrive and is charged per value, so the row
+ // group's charge is one number: the first set's 16068, three ranges of 40 and the second
+ // set's 8034, 24222 in all. A cap of it splices and one byte under it gives splicing up.
+ // Larger caps splice too, which a few of them check, among them 32216 to 32221. Those gave
+ // splicing up when a later set was sized from the one before it with no room for its ranges.
+ // The second set is not sized at all, so its byte child grows to 2000 bytes for its two
+ // 1000-byte values, where sizing it from the first set would have reserved 8000.
+ withTempDir { dir =>
+ val (path, _) = writeLaterSetFile(dir, lastTwoBytes = 1000)
+ val prefix = Substring(BoundReference(0, StringType, nullable = true), Literal(1), Literal(4))
+ val keys = Seq("0009", "0010", "0011", "0012", "0014", "0017").map(Literal(_))
+ def readUnder(cap: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, Seq(In(prefix, keys)), cap, batchSize = 4)
+ assertSpliceCharge(24222L, rows = 6)(readUnder)
+ val (_, uncapped) = readUnder("64MB")
+ Seq(32215L, 32216L, 32221L, 32222L).foreach { cap =>
+ val (rows, bytes) = readUnder(s"${cap}b")
+ assert(rows == 6 && bytes == uncapped,
+ s"a cap of $cap must keep the six rows and splicing; got $rows rows")
+ }
+ val (children, reader) = readAllWith(path, Seq("k", "v"),
+ createFilter(Seq(In(prefix, keys)), laterSetSchema),
+ (b, _) => ParquetTestAccess.byteChildCapacity(
+ b.column(0).asInstanceOf[WritableColumnVector]),
+ capacity = 4)
+ reader.close()
+ assert(children.drop(4).distinct == Seq(2000),
+ s"the second set's byte child must grow as values arrive; got $children")
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ assertSpliceCharge(166L, rows = 7) { cap =>
+ readUnderCap(path, schema, storageFilters, cap, batchSize = 16)
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warnings) = withLogged(partialReadWarning) {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(warnings == 0,
+ s"the footer check must get there before the exception path; got $warnings")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warnings) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(warnings == 0, "the footer check must decline the row group rather than the read " +
+ s"discovering it; got $warnings")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize = 64L * 1024 * 1024,
+ options = Map(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT -> "100"))
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ // The rows a reader hands back before it throws, what it throws, and, for the filtering
+ // reader, whether its scratch vectors were replaced on the way and the old ones freed. Both
+ // read off heap, where an old scratch vector left open would leak native memory.
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable, Boolean) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(true, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(true, 16, storageFilter)
+ }
+ def scratch: Seq[WritableColumnVector] = reader match {
+ case filtering: LateMaterializationParquetRecordReader =>
+ ParquetTestAccess.keyScratchVectors(filtering)
+ case _ => Nil
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val scratchBefore = scratch
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ val replacedAndFreed = scratch.zip(scratchBefore).exists { case (now, before) =>
+ (now ne before) && address(before) == 0
+ }
+ (read.toSeq, error, replacedAndFreed)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError, _) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError, scratchReplaced) =
+ readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ // A failed decode can leave the scratch vectors grown. Phase 2 reads the same pages into
+ // vectors of its own, so the reader replaces them with capacity-sized ones rather than keep
+ // the grown ones for the rest of the split.
+ assert(scratchReplaced,
+ "the decoding fallback must replace the scratch vectors and free the old ones")
+
+ // A kill that shows only at the decoding catch, after the per-chunk check has passed, is
+ // seen there. Without that catch's own check the row group would fall back and phase 2
+ // would fail on the page instead.
+ val killedAtTheCatch = spy(TaskContext.empty())
+ doAnswer((_: InvocationOnMock) => Boolean.box(Thread.currentThread().getStackTrace
+ .exists(_.getMethodName.startsWith("rethrowIfMustPropagate"))))
+ .when(killedAtTheCatch).isInterrupted()
+ val (_, killed, _) = TaskContext.withTaskContext(killedAtTheCatch) {
+ readUntilFailure(keyAtLeastFilter(500L))
+ }
+ assert(killed.isInstanceOf[SparkException] && killed.getMessage.contains("task kill") &&
+ killed.getCause != null && killed.getCause.getClass == plainError.getClass,
+ s"the kill must go out from the decoding catch, around the decoding error; got $killed")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+
+ test("a key vector that cannot grow, or a codec's InternalError, gives the row group up") {
+ // Both are fatal-looking errors that decoding the key pages can raise, and the reader falls
+ // back on both. A scratch vector that cannot grow is phase 1's own, and phase 2 then reads
+ // what a plain read reads. An `InternalError` from decoding is how a codec like hadoop-lzo
+ // reports a corrupt page, which the plain read of the same pages meets again. Raised from the
+ // filter instead, the first goes out as its OutOfMemoryError and the second in an
+ // `UnknownError`. Off heap, the scratch vectors the failed decode used must be freed, and the
+ // second row group's phase 1 runs over the ones that replace them.
+ withTempDir { dir =>
+ val rows = (0L until 40L).map(i => (i, i.toString))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "20"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(20L, 20L),
+ "the fixture must be two row groups of 20 rows")
+ Seq(
+ new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve")),
+ new InternalError("codec")).foreach { error =>
+ var failedScratch: Seq[WritableColumnVector] = Nil
+ // Only the first row group's key reader fails.
+ val failing: KeyReaderHook.OnKeyReader = (reader, keyReader) => {
+ val scratch = ParquetTestAccess.keyScratchVectors(
+ reader.asInstanceOf[LateMaterializationParquetRecordReader])
+ if (failedScratch.nonEmpty) {
+ // Checked before phase 1 writes into them, since a freed one would crash the JVM.
+ assert(scratch.forall(v => !failedScratch.exists(_ eq v) && address(v) != 0),
+ "the second row group must decode into new scratch vectors")
+ keyReader
+ } else {
+ failedScratch = scratch
+ val failingReader = spy(keyReader)
+ doThrow(error).when(failingReader).readBatch(anyInt(), any(), any(), any())
+ failingReader
+ }
+ }
+ val ((result, reader), warnings) = try {
+ withLogged(decodingGivenUp) {
+ readAll(path, keyAtLeastFilter(30L), useOffHeap = true, onKeyReader = failing)
+ }
+ } catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case e: VirtualMachineError => fail(s"the reader must fall back on $error; got $e", e)
+ }
+ try {
+ assert(result == rows.take(20) ++ rows.drop(30),
+ s"the first row group must come back whole and the second filtered on $error; got " +
+ s"${result.map(_._1)}")
+ assert(warnings == 1, s"the decoding fallback must report $error; got $warnings")
+ assert(failedScratch.forall(v => address(v) == 0),
+ s"the scratch vectors of the failed decode must be freed on $error")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("a key value the filter does not read and cannot be decoded gives splicing up") {
+ // The filter need not read every key on every row, as `coalesce(a, b)` reads `b` only where
+ // `a` is null. The copy into the survivor buffer reads every key, so it can meet a value no
+ // plain read decodes. A dictionary-encoded column decodes a value only when it is read, so a
+ // dictionary that holds fewer values than its ids point at is such a value. Copying it gives
+ // splicing up for the row group, and phase 2 reads the key columns the way a plain read does.
+ // So the read answers where a plain one does, and the filter still applies.
+ withTempDir { dir =>
+ // Two copies of each `b`, so parquet keeps its dictionary. Cut to 64 values, it leaves `b`
+ // undecodable from row 128 on. `a` is null only before that, so the filter never reads a
+ // damaged `b`.
+ val rows = (0 until 200).map { i =>
+ (if (i < 100 && i % 5 == 0) null else s"a_$i", s"b_${i / 2}", i)
+ }
+ val path = writeSingleParquetFile(dir, rows.toDF("a", "b", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE, dictionary = true)
+ assert(encodingsOf(path, "b").exists(_.usesDictionary),
+ s"the fixture's whole point; got ${encodingsOf(path, "b")}")
+ shrinkDictionary(path, column = "b", entries = 64)
+ val columns = Seq("a", "b", "v")
+ intercept[ArrayIndexOutOfBoundsException] {
+ readAllWith(path, columns, null, (b, i) => b.column(1).getUTF8String(i).toString)._2
+ .close()
+ }
+
+ val schema = StructType(Seq(
+ StructField("a", StringType), StructField("b", StringType), StructField("v", IntegerType)))
+ val coalesced = Coalesce(Seq(BoundReference(0, StringType, nullable = true),
+ BoundReference(1, StringType, nullable = true)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(coalesced, Literal.create("a_15", StringType))), schema)
+ val ((result, reader), givenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, columns, filter, (b, i) => b.column(2).getInt(i), capacity = 16)
+ }
+ try {
+ // Keeps rows 150-199 among others, whose `b` is damaged.
+ val expected = rows.filter { case (a, b, _) => Option(a).getOrElse(b) >= "a_15" }.map(_._3)
+ assert(expected.contains(150), "the filter must keep a row whose `b` is damaged")
+ assert(result == expected, s"got ${result.size} rows, expected ${expected.size}")
+ assert(givenUp == 1,
+ "the copy must have failed and given splicing up once, or this proves nothing; " +
+ s"logged $givenUp")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry reads the damaged row group whole over the pushed data filter's ranges, and only
+ // that one, since parquet keeps a column index store per block. What it must not give up is
+ // the pushed data filter's own page pruning, which a plain read of the same query keeps, nor
+ // the other row groups' narrowing. Dropping either would read more than it needs to.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warnings) = withLogged(partialReadWarning)(read(evenKeys))
+ assert(warnings == 1,
+ "the read must have discovered the damaged index, or this test proves nothing; got " +
+ warnings)
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
Review Comment:
**[P3, nonblocking] Assert exact multiplicity in the damaged-index fallback test.**
Could this compare the exact ordered output against `plain.filter(k => (k >= 101L && k <= 200L) || k % 2 == 0)`? The current membership and size assertions can pass if a matching row is duplicated while another nonmatching row is omitted. Exact equality would also pin multiplicity across the damaged-row-group retry.
This is a concrete coverage improvement, not evidence of a production duplication bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4097170674
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -151,6 +151,60 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
}
}
+ /**
+ * Splits `afterScanFilters` into conjuncts the file format can evaluate at the storage layer for
+ * late materialization (returned as the first element) and the remaining filters that stay as a
+ * post-scan FilterExec (returned as the second element).
+ *
+ * Everything is checked here, statically, so the runtime never silently loses a filter. Two of
+ * the conditions are per scan, and failing either leaves every filter in the plan:
+ * - The storage-filter pushdown SQL conf is on.
+ * - [[FileFormat.supportBatch]] holds for the schema the reader will see,
+ * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives
+ * its own vectorized-read decision from. Late materialization needs a batch read, so this asks
+ * about batch support rather than naming a format.
+ *
+ * The rest are per conjunct, and one that fails any of them stays in the post-scan Filter:
+ * - It is deterministic. A reader evaluates the predicate without
+ * `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a
+ * `Nondeterministic` expression, so a non-deterministic conjunct would fail at task time.
+ * - It references at least one column, and every column it references is a projected data
+ * column. A reference to something the scan does not read cannot be evaluated by the reader.
+ * - [[FileFormat.supportsStorageFilter]] accepts it. That is where the expression shapes and
+ * column types a reader can evaluate live, so this method names neither a format nor a type.
+ *
+ * One last condition is on the set that survives: at least one projected data column must be left
+ * for the reader to prune. A scan that projects nothing but the filter's own key columns reads
+ * the same columns for the same rows either way, since the reader has to read a key column to
+ * evaluate the filter on it, so pushing can only add the cost of evaluating the predicate outside
+ * the generated code. Extraction is dropped entirely in that case.
+ */
+ private def extractStorageFilters(
+ afterScanFilters: ExpressionSet,
+ fsRelation: HadoopFsRelation,
+ readDataColumns: Seq[Attribute],
+ outputDataSchema: StructType): (Seq[Expression], ExpressionSet) = {
+ val sparkSession = fsRelation.sparkSession
+ val sqlConf = sparkSession.sessionState.conf
+ if (!sqlConf.parquetStorageFilterPushdownEnabled) return (Nil, afterScanFilters)
+ val resultSchema = StructType(fsRelation.partitionSchema.fields ++ outputDataSchema.fields)
+ if (!fsRelation.fileFormat.supportBatch(sparkSession, resultSchema)) {
+ return (Nil, afterScanFilters)
+ }
+
+ val dataAttrs = AttributeSet(readDataColumns)
+ val (eligible, rest) = afterScanFilters.partition { expr =>
Review Comment:
Real, and reproduced rather than reasoned about: with the gate ablated, a `CAST(s AS BIGINT)` key plus a `WHERE kind = 'num'` guard over a non-numeric row in the same row group fails with `CAST_INVALID_INPUT`.
`ParquetStorageFilter.isSupportedStorageFilter` now requires the hash's children to pass `ExprUtils.canEvaluateUnconditionally`, which is the whitelist master already uses for this hazard class. That covers `evalAllMissing` too, since it evaluates the same expression. You were right about `throwable`: it is opt-in and `Cast` does not set it, which `ExprUtils`' own scaladoc calls out.
Tested on the gate and end to end, with the offending row in the same row group as matching ones so no statistic prunes it away. c255f88
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its "
+ + "filter kept in row group %d of %s, but the file was written without a page "
+ + "index. Set %s=false to read this file.",
+ finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(),
+ SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new UnsupportedFileReadException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it --
+ // which can make the row group's contribution negative, and that is the truth about it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
Review Comment:
Gone with the design change in the `DataSourceUtils` thread: there is no `UnsupportedFileReadException` and no case for it in the shared classifier any more.
The cast error you describe cannot reach the reader either, since such an expression is no longer offered. What is left is that an internal error in this reader is swallowed under `ignoreCorruptFiles` the way any other reader's is, which is now the same behaviour as the rest of the read path rather than a special case. c255f88
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its "
+ + "filter kept in row group %d of %s, but the file was written without a page "
+ + "index. Set %s=false to read this file.",
+ finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(),
+ SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new UnsupportedFileReadException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it --
+ // which can make the row group's contribution negative, and that is the truth about it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
Review Comment:
Confirmed in the code: `ParquetReadState.constructRanges` materializes a `List<RowRange>` per leaf per row group, so scattered survivors cost about 40 bytes a range times the leaf count, and phase 2 drives one reader per leaf.
It is counted now and weighed against the same cap as the survivor buffer, in two rungs: drop the buffer first, and give the filter up for that row group if the ranges alone still do not fit. There is a second check right after phase 1, for a row group whose survivors fit in a single accumulator and are therefore never weighed inside the loop.
The better answer is a follow-up in the description: coarsen the ranges instead of giving the filter up. Closing a gap shorter than the smallest page row count of the columns phase 2 reads costs no extra bytes, since no page fits wholly inside such a gap. c255f88
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
Review Comment:
Fixed by not letting it escape. The phase-2 read is wrapped, and on `MissingOffsetIndexException` the filter is given up for that row group and the read retried over `pushedFilterRanges`, which is what a plain scan reads. The file is latched, so later row groups skip phase 1 rather than evaluate a filter they cannot use.
So there is no exception left to be misleading, and nothing for `ignoreCorruptFiles` to swallow. The conf doc says what it costs: the first row group the reader tries it on also pays a key-column read, because a missing index is only reported by attempting the read. c255f88
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its "
+ + "filter kept in row group %d of %s, but the file was written without a page "
+ + "index. Set %s=false to read this file.",
+ finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(),
+ SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new UnsupportedFileReadException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it --
+ // which can make the row group's contribution negative, and that is the truth about it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
Review Comment:
Right, `SQLMetrics.scala` has `if (v >= 0)`. The second key read is now charged into that row group's own phase-2 bytes, so the per-row-group number is honest, and the comment says what the clamp then does: a row group that read more than the baseline contributes nothing rather than subtracting.
A test asserts the charge, by running the same file and filter with the cap high and low and requiring the low arm to report less avoided. c255f88
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:
##########
@@ -230,6 +274,40 @@ class ParquetFileFormat
val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead
val archiveFormatEnabled = parquetOptions.archiveFormatEnabled
+ // Extraction has already removed these conjuncts from the post-scan Filter, so nothing else in
+ // the plan would apply them: anything that stops the reader from honoring them fails loudly
+ // rather than dropping them. `enableVectorizedReader` is one such thing, and it is recomputed
+ // from the live session conf when the RDD is built, so a flip of
+ // spark.sql.parquet.enableVectorizedReader (or the nested-column variant) after planning lands
+ // here.
+ val storageFilterOpt: Option[ParquetStorageFilter] = if (storageFilters.isEmpty) {
+ None
+ } else if (!enableVectorizedReader) {
+ throw new UnsupportedFileReadException(
Review Comment:
Fixed: the messages name the expression by its node rather than printing it, so a prepared bloom cannot render as megabytes of hex. The `require` in `FileFormat.buildReaderWithStorageFilters` is gone with the model change, since that default is now a working implementation rather than a guard. c255f88
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala:
##########
@@ -269,7 +269,15 @@ object DataSourceUtils extends PredicateHelper {
QueryExecutionErrors.sparkUpgradeInWritingDatesError(format, config)
}
+ /**
+ * Whether `ignoreCorruptFiles` may swallow this failure and skip the rest of its file.
+ *
+ * [[UnsupportedFileReadException]] is excluded because it does not report a corrupt file: it says
+ * the file cannot support a read the plan depends on. Skipping the rest of such a file would drop
+ * rows that are perfectly readable, and would do it silently.
+ */
def shouldIgnoreCorruptFileException(e: Throwable): Boolean = e match {
+ case _: UnsupportedFileReadException => false
Review Comment:
Taken, and it is the biggest change in this round. Thank you for pushing on it.
The conjunct is not moved out of the plan at all now, it is copied: it stays in the post-scan `Filter` as well, so the reader is free to give a file or a row group up with no correctness argument attached. `UnsupportedFileReadException` and its case in the shared classifier are gone with it, and so is the hard failure on a file written without a page index.
Two things settled it beyond the fallback you asked for. It is v1's own taxonomy already, where only partition filters leave the plan because partition pruning is exact, while the data filters handed to Parquet stay, and `PushDownUtils` draws the same line from the other side in DSv2. And the bloom is never built for nothing: the `Filter` uses it whatever the scan managed to do, and both sides read the same bytes because the subquery is executed once.
The price is evaluating the conjunct a second time for the rows the scan emits. That falls on what the filter kept rather than on what it discarded, and anyone turning this conf on is doing it because the filter is selective, so it is a small price. c255f88
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
Review Comment:
You are right, and the description was wrong. `clipParquetGroupFields` keeps a field that is missing from the physical file, so `nonKeyFieldCount` is always greater than zero once the planner rejects all-keys projections. Both the description and the class comment are corrected.
The branch stays, since the reader should not depend on a planner rule to be correct, and the comment now says what is true: nothing in production reaches it, only a test that drives the reader directly. c255f88
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its "
+ + "filter kept in row group %d of %s, but the file was written without a page "
+ + "index. Set %s=false to read this file.",
+ finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(),
+ SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new UnsupportedFileReadException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it --
+ // which can make the row group's contribution negative, and that is the truth about it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
+ }
+ }
+ }
+ remaining -= num;
+ }
+
+ if (accumulate) {
+ finalizePartialAccumulators();
+ }
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ // Assigned before the loop on purpose: an allocation failure part way through then leaves the
+ // vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows
+ * each.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, pushing them onto their
+ * queues once full. All key columns advance in lockstep, which is what keeps the queues aligned.
+ *
+ * <p>Returns false when the buffered survivors have passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} and this row group has
+ * given splicing up, in which case it has already released what it held and the caller must stop
+ * calling this.
+ */
+ private boolean appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ long valueBytes = 0L;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ // Measured on the destination: a dictionary-encoded source has no length of its own, since
+ // its values are read through the dictionary.
+ if (keyVariableLength[i]) valueBytes += dst.getArrayLength(dstRow);
+ }
+ }
+ splicedBytes += keyFixedBytesPerRow + valueBytes;
Review Comment:
Follow-up, now listed in the description as accounting the buffer through a `MemoryConsumer` whose `spill()` gives splicing up.
The 2x you describe is acknowledged rather than hidden: the conf doc says what is counted, the buffered values and their per-row overhead, not the backing arrays a column vector may grow beyond that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4108708237
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Asked this way rather than up front, from the footer. Parquet resolves the index over the
+ // paths current at its first lookup for the block, which without a pushed data filter is
+ // the non-key columns alone, so a footer walk over the projection is both stricter than the
+ // read and blind to an index that is claimed but unreadable. It is also where the throw
+ // costs least: it lands before any data page is read.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
Review Comment:
**This extends trust in the Parquet OffsetIndex to bloom-only queries.** With no pushed data filter, or one that prunes no page, the plain read path never consults the offset index, because `readNextFilteredRowGroup` falls back to `readNextRowGroup`. Here, phase 2 reads the non-key columns through `internalReadFilteredRowGroup` for every row group the bloom narrows, and the page `firstRowIndex` values come from the offset index.
Consider a file whose offset index is present but wrong. An example is a writer that split repeated records across pages while writing a page index (apache/arrow-rs#3680). Such a file is read correctly today, but with this feature the non-key values would be paired with the wrong spliced keys. The key itself is correct, so the post-scan `Filter` cannot catch it. The only escape hatch is `parquet.filter.columnindex.enabled=false`, and a user with no pushed predicate has no reason to set it. It would be good to at least document this in the conf doc.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Asked this way rather than up front, from the footer. Parquet resolves the index over the
+ // paths current at its first lookup for the block, which without a pushed data filter is
+ // the non-key columns alone, so a footer walk over the projection is both stricter than the
+ // read and blind to an index that is claimed but unreadable. It is also where the throw
+ // costs least: it lands before any data page is read.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Reading {} without page-level storage filtering: reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs. Row groups the filter empties are still skipped whole", e,
+ MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ // The retry only avoids the same wall because a store missing one column's offset index
+ // reports no column index either, so these ranges cover the whole block and parquet reads
+ // it without consulting an index. That is three parquet internals deep, so it is checked:
+ // a release that changes any of them should fail here rather than throw from the read.
+ if (baselineRows != blockRowCount) {
+ throw new IllegalStateException(String.format(
+ "Cannot read row group %d of %s without an offset index: the pushed filter selects "
+ + "%d of %d rows, so a plain read of the block is not what it asks for",
+ blockIdx, lateMatReader.getFile(), baselineRows, blockRowCount));
+ }
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up: it read what a plain scan
+ // reads, so the saving is a certain zero.
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ long phase2Bytes = compressedBytesForRowRanges(blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = lateMatReader.getRowRanges(blockIdx);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private long compressedBytesForRowRanges(
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore =
+ wholeBlock ? null : lateMatReader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ // The store answers null, rather than throwing, for a path it was not built with. That can
+ // only mean the block's store was built while a narrower schema was requested than this
+ // walk asks about, an ordering bug rather than a property of the file, and the footer tells
+ // the two apart. It is reported rather than thrown: this walk only produces a counter, and
+ // `ignoreCorruptFiles` turns any exception from a reader into a silently truncated file, so
+ // a byte metric must not be able to change the answer.
+ if (chunk.getOffsetIndexReference() != null && !loggedMissingStoreEntry) {
+ loggedMissingStoreEntry = true;
+ LOG.warn("Undercounting the storage filter's avoided bytes for {}: column "
+ + path.toDotString() + " of row group " + blockIndex + " has an offset index the "
+ + "block's column index store was not built with",
+ MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ }
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below: the bytes buffered for
+ // splicing, and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = storageFilter.test(keyScratchBatch.getRow(r));
+ } catch (RuntimeException e) {
+ if (!storageFilter.isEvaluationError(e)) throw e;
Review Comment:
**Fail-open covers only non-internal `SparkThrowable`s.** `isEvaluationError` (`ParquetStorageFilter.scala:157`) returns `false` for any plain JDK `RuntimeException`, so the exception is rethrown here, and also from `evalAllMissing` during `initialize`. Some built-in key expressions admitted by `InjectRuntimeFilter.isSimpleExpression` and `canEvaluateInTheReader` throw raw JDK exceptions, e.g. `timestamp_seconds(decimal_col)` (`ArithmeticException` from `longValueExact`) or `format_string('%c', cp)` (`IllegalFormatCodePointException`).
Take `... JOIN d ON timestamp_seconds(a.d) = d.ts WHERE a.kind = 'num'`, where the offending rows have `kind = 'text'`. The pushed filter prunes only at page granularity, so phase 1 evaluates the bloom on those rows too:
- `spark.sql.files.ignoreCorruptFiles=false`: the task fails, while the same query succeeds with the feature off. That contradicts the contract documented on `FileFormat.buildReaderWithStorageFilters` ("A reader must not fail the query for that").
- `spark.sql.files.ignoreCorruptFiles=true`: `FileScanRDD` treats the `RuntimeException` as a corrupt file and silently skips the rest of the file, so join rows go missing. Without the feature, the same exception would come from `FilterExec`, outside that `try`, and fail the query.
Since the conjunct stays in the post-scan `Filter`, failing open on any non-fatal exception from `test()`/`evalAllMissing()` looks safe. A genuine error is raised again by `FilterExec` on the same row, in plan order.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -391,15 +574,54 @@ public void enableReturningBatches() {
* Advances to the next batch of rows. Returns false if there are no more.
*/
public boolean nextBatch() throws IOException {
+ releasePublishedKeyVectors();
for (ParquetColumnVector vector : columnVectors) {
vector.reset();
}
+ // Zeroed before the terminal checks below, so a terminal call cannot leave a spliced batch
+ // pointing at the key vectors just released, which off heap is freed memory.
columnarBatch.setNumRows(0);
+ if (hitEndOfData) return false;
if (rowsReturned >= totalRowCount) return false;
checkEndOfRowGroup();
+ if (hitEndOfData) return false;
int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned);
- for (ParquetColumnVector cv : columnVectors) {
+ // A spliced row group takes its key slots from the survivor queues, so phase 2 skipped them.
+ // Everything else read every projected column: a plain read with no storage filter at all, or a
+ // row group that gave splicing up, whose batch needs its key slots put back to the persistent
+ // vectors a previous row group rewrote. `spliceCurrentRowGroup` stays false without a storage
+ // filter, so it answers for all three.
+ if (spliceCurrentRowGroup) {
+ publishSurvivorKeyVectors(num);
+ } else if (spliceBatchColumns != null) {
+ System.arraycopy(persistentBatchColumns, 0, spliceBatchColumns, 0, spliceBatchColumns.length);
+ }
+ readPersistentColumns(num, /* skipKeySlots= */ spliceCurrentRowGroup);
+ // If needed, compute row indexes within a file. The row-index column is identified by name
+ // (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references,
+ // so its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ if (rowIndexGenerator != null) {
Review Comment:
This holds only for the synthetic metadata column. `ParquetRowIndexUtil` finds the row-index slot purely by name (`_tmp_metadata_row_index`), and that name is not reserved for data columns; `ParquetFileMetadataStructRowIndexSuite` even reads such a column. A real `BIGINT` data column with that name is in `readDataColumns`, so it can become a storage-filter key.
Its output then depends on how each row group was read:
- Spliced row groups emit the file values from the survivor queues.
- Row groups that gave up the filter or splicing emit the persistent vector, which `populateRowIndex` has overwritten with row indexes.
- If the column is declared but missing from the file, the reader materializes row indexes while the filter is evaluated on `null` (the `XxHash64` seed), so `evalAllMissing` can skip the whole file.
This is already broken without the feature (SPARK-40059), but now the result also varies with the conf and between row groups of one file. Excluding the row-index column from key slots, or rejecting it in `storageFiltersFor`, would keep it consistent.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:
##########
@@ -190,6 +192,53 @@ class ParquetFileFormat
filters: Seq[Filter],
options: Map[String, String],
hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow] = {
+ buildParquetReader(
+ sparkSession, dataSchema, partitionSchema, requiredSchema, filters, Nil, options, hadoopConf,
+ Map.empty)
+ }
+
+ /**
+ * The conf that turns this on is read here rather than in the planner, the way `supportBatch`
+ * reads its own confs, so a Parquet-named conf does not decide for a format that is not Parquet.
+ *
+ * Subclasses answer false on purpose, even though they inherit this reader: a subclass may
+ * customize reading by overriding `buildReaderWithPartitionValues`, and a scan with storage
+ * filters routes through `buildReaderWithStorageFilters` instead, which would silently bypass
+ * whatever the subclass does.
+ */
+ override def supportsStorageFilterPushdown(sparkSession: SparkSession): Boolean =
+ getSqlConf(sparkSession).parquetStorageFilterPushdownEnabled &&
+ getClass == classOf[ParquetFileFormat]
+
+ override def supportsStorageFilter(expr: Expression): Boolean =
+ ParquetStorageFilter.isSupportedStorageFilter(expr)
+
+ override def buildReaderWithStorageFilters(
Review Comment:
The subclass guard (`getClass == classOf[ParquetFileFormat]`) exists only in `supportsStorageFilterPushdown`, which is checked at planning time. Every subclass inherits this override, and it always returns `Some(<base Parquet reader>)`.
So if an extension rule copies a scan onto a subclass relation, or builds a `FileSourceScanExec` with `storageFilters`, `inputRDD` silently bypasses the subclass's `buildReaderWithPartitionValues` (for example deletion vectors or column mapping). I didn't find an in-repo path that does this. Still, returning `None` here for subclasses would be a cheap defense, since `inputRDD` already falls back when it gets `None`.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -409,23 +631,250 @@ public boolean nextBatch() throws IOException {
}
cv.assemble();
}
- // If needed, compute row indexes within a file.
- if (rowIndexGenerator != null) {
- rowIndexGenerator.populateRowIndex(columnVectors, num);
- }
+ }
- rowsReturned += num;
+ /** Publishes {@code num} rows as the current batch. */
+ private void finishBatch(int num) {
columnarBatch.setNumRows(num);
+ rowsReturned += num;
numBatched = num;
batchIdx = 0;
- return true;
+ }
+
+ /**
+ * Points the batch's key slots at one survivor key vector each, which is what publishes them: the
+ * same {@link ColumnarBatch} is handed out every time, over an array it holds by reference. The
+ * queues keep owning those vectors until the next batch releases them, so a phase-2 read that
+ * throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void publishSurvivorKeyVectors(int num) {
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ if (keyVectorQueues[i].isEmpty()) {
+ // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop
+ // is driven by that same count. Named rather than left to NoSuchElementException.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to "
+ + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num));
+ }
+ }
+ keyVectorsPublished = true;
+ // Queue k holds the survivors of key column k, and `keyColumnIndices[k]` is the batch slot that
+ // key column sits in, so the pairing is read off rather than re-derived from slot order. Only
+ // key slots are touched, since a non-key slot never holds anything but its persistent vector.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int k = 0; k < keyIndices.length; k++) {
+ spliceBatchColumns[keyIndices[k]] = keyVectorQueues[k].peekFirst();
Review Comment:
Publishing survivors by swapping `ColumnVector` objects into the batch array, and freeing them at the next `nextBatch()`, is what requires several extra pieces:
- `spliceBatchColumns`/`persistentBatchColumns`;
- the `arraycopy` restore;
- the special `close()` path;
- the `resultBatch()` contract change.
It also changes what happens to a caller that keeps `batch.column(k)` across `nextBatch()`, as `DataSourceReadBenchmark` does (without splicing). That caller now reads freed memory: off-heap it is a SIGSEGV because `close()` zeroes `data`, and on-heap it is an NPE. Before, it would have read stale values.
Batches and accumulator chunks are both `capacity`-sized and aligned, so a simpler option exists. At emit, copy chunk k into the already-allocated persistent key vector with the existing `keyCopiers`, then close the chunk right away. The persistent vector needs `setDictionary(null)` first, since it may carry a dictionary from a non-spliced row group. The batch array is then never rewritten, and `close()` can go back to `columnarBatch.close()`.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -409,23 +631,250 @@ public boolean nextBatch() throws IOException {
}
cv.assemble();
}
- // If needed, compute row indexes within a file.
- if (rowIndexGenerator != null) {
- rowIndexGenerator.populateRowIndex(columnVectors, num);
- }
+ }
- rowsReturned += num;
+ /** Publishes {@code num} rows as the current batch. */
+ private void finishBatch(int num) {
columnarBatch.setNumRows(num);
+ rowsReturned += num;
numBatched = num;
batchIdx = 0;
- return true;
+ }
+
+ /**
+ * Points the batch's key slots at one survivor key vector each, which is what publishes them: the
+ * same {@link ColumnarBatch} is handed out every time, over an array it holds by reference. The
+ * queues keep owning those vectors until the next batch releases them, so a phase-2 read that
+ * throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void publishSurvivorKeyVectors(int num) {
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ if (keyVectorQueues[i].isEmpty()) {
+ // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop
+ // is driven by that same count. Named rather than left to NoSuchElementException.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to "
+ + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num));
+ }
+ }
+ keyVectorsPublished = true;
+ // Queue k holds the survivors of key column k, and `keyColumnIndices[k]` is the batch slot that
+ // key column sits in, so the pairing is read off rather than re-derived from slot order. Only
+ // key slots are touched, since a non-key slot never holds anything but its persistent vector.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int k = 0; k < keyIndices.length; k++) {
+ spliceBatchColumns[keyIndices[k]] = keyVectorQueues[k].peekFirst();
+ }
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (java.util.ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
missingColumns = new HashSet<>();
for (ParquetColumn column : CollectionConverters.asJava(parquetColumn.children())) {
checkColumn(column);
}
+ if (storageFilter != null) {
+ initializeLateMaterialization();
+ }
+ }
+
+ /**
+ * Sets the storage filter for late materialization. Must be called before {@link #initialize};
+ * {@link #initializeLateMaterialization()} then inspects the per-file schema and decides whether
+ * splicing engages.
+ *
+ * <p>It does not engage in one real case: every key column is missing from this physical file
+ * under schema evolution. The predicate is then rewritten with each missing key replaced by the
+ * constant the reader materializes for it (its existence DEFAULT, else null) and evaluated once.
+ * True keeps the file unfiltered, false or null skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by
+ * {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}, so a
+ * violation of it here is a planner bug and is asserted rather than handled. A file the reader
+ * simply cannot prune is a different matter: it reads it the way a plain scan would.
+ */
+ public void setStorageFilter(ParquetStorageFilter storageFilter) {
+ this.storageFilter = storageFilter;
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (!configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true)) {
Review Comment:
Turning the whole feature off here also drops the bloom's whole-row-group skipping, which needs no page index at all: `readFilteredRowGroup` with whole-block ranges goes straight to `internalReadRowGroup`. The reader already has that mode (`fileHasNoOffsetIndex`), so "page index unusable" is now implemented in two places.
When the conf is false, you could start with `fileHasNoOffsetIndex = true` and use `RowRanges.createSingle(blockRowCount)` in phase 0 and in `recordFileSkipped`. The latter is needed because `ParquetFileReader.getRowRanges(int)` ignores `useColumnIndexFilter()`, unlike `getFilteredRecordCount()`. This stays consistent with `totalRowCount = getRecordCount()` in that case.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -409,23 +631,250 @@ public boolean nextBatch() throws IOException {
}
cv.assemble();
}
- // If needed, compute row indexes within a file.
- if (rowIndexGenerator != null) {
- rowIndexGenerator.populateRowIndex(columnVectors, num);
- }
+ }
- rowsReturned += num;
+ /** Publishes {@code num} rows as the current batch. */
+ private void finishBatch(int num) {
columnarBatch.setNumRows(num);
+ rowsReturned += num;
numBatched = num;
batchIdx = 0;
- return true;
+ }
+
+ /**
+ * Points the batch's key slots at one survivor key vector each, which is what publishes them: the
+ * same {@link ColumnarBatch} is handed out every time, over an array it holds by reference. The
+ * queues keep owning those vectors until the next batch releases them, so a phase-2 read that
+ * throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void publishSurvivorKeyVectors(int num) {
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ if (keyVectorQueues[i].isEmpty()) {
+ // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop
+ // is driven by that same count. Named rather than left to NoSuchElementException.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to "
+ + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num));
+ }
+ }
+ keyVectorsPublished = true;
+ // Queue k holds the survivors of key column k, and `keyColumnIndices[k]` is the batch slot that
+ // key column sits in, so the pairing is read off rather than re-derived from slot order. Only
+ // key slots are touched, since a non-key slot never holds anything but its persistent vector.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int k = 0; k < keyIndices.length; k++) {
+ spliceBatchColumns[keyIndices[k]] = keyVectorQueues[k].peekFirst();
+ }
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (java.util.ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
missingColumns = new HashSet<>();
for (ParquetColumn column : CollectionConverters.asJava(parquetColumn.children())) {
checkColumn(column);
}
+ if (storageFilter != null) {
+ initializeLateMaterialization();
+ }
+ }
+
+ /**
+ * Sets the storage filter for late materialization. Must be called before {@link #initialize};
+ * {@link #initializeLateMaterialization()} then inspects the per-file schema and decides whether
+ * splicing engages.
+ *
+ * <p>It does not engage in one real case: every key column is missing from this physical file
+ * under schema evolution. The predicate is then rewritten with each missing key replaced by the
+ * constant the reader materializes for it (its existence DEFAULT, else null) and evaluated once.
+ * True keeps the file unfiltered, false or null skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by
+ * {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}, so a
+ * violation of it here is a planner bug and is asserted rather than handled. A file the reader
+ * simply cannot prune is a different matter: it reads it the way a plain scan would.
+ */
+ public void setStorageFilter(ParquetStorageFilter storageFilter) {
+ this.storageFilter = storageFilter;
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (!configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true)) {
+ // That conf is the escape hatch for a file whose page index is wrong, so it has to turn this
+ // feature off whole rather than only its filtering. Phase 2 reads part of a row group through
+ // the offset index, which parquet consults whatever the conf says, and a wrong one there
+ // pairs a row's key with another row's values. The post-scan filter cannot catch that: the
+ // key it sees is the right one.
+ storageFilter = null;
+ return;
+ }
+ lateMatReader = reader.getUnderlyingReader();
+ if (lateMatReader == null) {
+ // Late materialization drives a ParquetFileReader directly, so without one the filter is
+ // simply not applied and the plain read path runs. The conjunct is in the post-scan filter as
+ // well, so the rows it would have dropped are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ // Resolve each key column's top-level ParquetColumn. Partition into present and missing (the
+ // latter can happen under schema evolution: a column is in the requested schema but not in this
+ // physical parquet file). For any non-primitive key we still bail; phase-1 reads only primitive
+ // leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> missingKeyLocalPositions = new ArrayList<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (missingColumns.contains(column)) {
+ missingKeyLocalPositions.add(i);
+ } else {
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // If any key column is missing from this file, rewrite the predicate to substitute the constant
+ // the reader will actually materialize for that column. That is the column's existence DEFAULT
+ // when it has one (ParquetColumnVector writes it into the output vector and marks the vector
+ // constant), otherwise null. Substituting null for a column that reads back as its default
+ // would filter on a value the scan never returns. The predicate must be evaluated against the
+ // substituted constant rather than skipped: null does not always mean false in a filter.
+ if (!missingKeyLocalPositions.isEmpty()) {
+ int[] missing = new int[missingKeyLocalPositions.size()];
+ Object[] missingValues = new Object[missingKeyLocalPositions.size()];
+ Object[] existenceDefaults =
+ ResolveDefaultColumns.existenceDefaultValues(sparkRequestedSchema);
+ for (int i = 0; i < missing.length; i++) {
+ missing[i] = missingKeyLocalPositions.get(i);
+ missingValues[i] = existenceDefaults[keyIndices[missing[i]]];
+ }
+ storageFilter = storageFilter.rewriteForMissingKeys(missing, missingValues);
Review Comment:
This re-derives what the reader materializes for a missing key column (the existence DEFAULT, else `null`) and rewrites the predicate with literals. That duplicates `ParquetColumnVector`'s own missing-column rule. Because the rewrite rebuilds `BloomFilterMightContain`, it also needs the per-task `rewrites` cache and the "values are a function of the positions" invariant. I found no mismatch today, but if the two rules ever diverge, the filter would silently evaluate a value the scan never returns.
An alternative:
- keep the predicate and `keyColumnIndices` unchanged;
- put the constant/all-null persistent vectors that `initBatch` already built into the key-scratch batch's missing slots (`reset()` is a no-op for them);
- skip missing keys when splicing.
`evalAllMissing` would then need to move after `initBatch`.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.SparkThrowable
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Literal, Predicate, XxHash64}
+import org.apache.spark.sql.catalyst.trees.TreePattern.PLAN_EXPRESSION
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.{BinaryType, BooleanType, ByteType, DataType, DateType, DayTimeIntervalType, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType, StringType, StructType, TimestampNTZType, TimestampType, TimeType, YearMonthIntervalType}
+
+/**
+ * Optional SQL metrics the reader updates while applying a [[ParquetStorageFilter]]. Every
+ * counter is scoped to what the storage filter added on top of a read of the same projection
+ * without one. All fields are nullable; a null field disables that metric.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums the rows those skips excluded, per skipped block the rows that
+ * survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it, and it is zero on an
+ * all-keys projection, which can avoid nothing.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned.
+ *
+ * The row counters' suffix says where a row was excluded, not what would have saved it: a row
+ * inside a kept row group is read as part of its page and dropped during decode, so page
+ * filtering did not save it. An all-keys projection has no page filtering at all, and
+ * [[rowsExcludedWithinRowGroup]] still counts every row the filter dropped.
+ */
+case class StorageFilterMetrics(
Review Comment:
All five fields are nullable, and `create` has default arguments. However, the only production path (`FileSourceScanLike.storageFilterMetrics` → `inputRDD` → `buildParquetReader`) always supplies all five, as the reader's own comment at `VectorizedParquetRecordReader.java:1002-1004` says. So several pieces exist only for tests:
- the ~10 null checks in the reader;
- the `needBytes` mode;
- the five `getOrElse(key, null)` in `ParquetFileFormat`.
Making the fields required, and dropping the `storageFilterMetrics = Map.empty` default on `FileFormat.buildReaderWithStorageFilters`, would simplify this. Tests would then pass real `SQLMetric`s.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -642,7 +656,26 @@ trait FileSourceScanLike extends DataSourceScanExec with SessionStateHelper {
} else {
None
}
- } ++ driverMetrics
+ } ++ storageFilterMetrics ++ driverMetrics
+
+ protected lazy val storageFilterMetrics: Map[String, SQLMetric] = if (storageFilters.nonEmpty) {
Review Comment:
This defines Parquet-specific metric names ("row groups", "page filtering") in the format-neutral `FileSourceScanLike`, and `ParquetFileFormat` now imports the scan's companion to look them up with `getOrElse(..., null)`. A second format that opts in (e.g. ORC) would show Parquet-named metrics it cannot fill.
The same principle this PR applies to the conf and to the expression checks (the format decides) suggests a `FileFormat` hook that returns the format's own metric map. That would keep the scan format-neutral and let the reader's metrics be non-optional. DSv2's `supportedCustomMetrics` / `SQLMetrics.createV2CustomMetric` is the precedent.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
Review Comment:
Minor, on `spark.sql.files.ignoreCorruptFiles` semantics: phase 1 decompresses and decodes every key page of a row group before any row of it is emitted. Page decompression is lazy in both paths, so the raw chunk read succeeds even when a page is corrupt. If a key page near the end of a row group is corrupt, the plain reader has already returned the batches before it, but this path returns none of that row group's rows. That differs from the documented "the contents that have been read will still be returned". The result shrinks rather than being wrong, so this may be acceptable, but it may deserve a note.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Asked this way rather than up front, from the footer. Parquet resolves the index over the
+ // paths current at its first lookup for the block, which without a pushed data filter is
+ // the non-key columns alone, so a footer walk over the projection is both stricter than the
+ // read and blind to an index that is claimed but unreadable. It is also where the throw
+ // costs least: it lands before any data page is read.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Reading {} without page-level storage filtering: reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs. Row groups the filter empties are still skipped whole", e,
+ MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ // The retry only avoids the same wall because a store missing one column's offset index
+ // reports no column index either, so these ranges cover the whole block and parquet reads
+ // it without consulting an index. That is three parquet internals deep, so it is checked:
+ // a release that changes any of them should fail here rather than throw from the read.
+ if (baselineRows != blockRowCount) {
+ throw new IllegalStateException(String.format(
+ "Cannot read row group %d of %s without an offset index: the pushed filter selects "
+ + "%d of %d rows, so a plain read of the block is not what it asks for",
+ blockIdx, lateMatReader.getFile(), baselineRows, blockRowCount));
+ }
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up: it read what a plain scan
+ // reads, so the saving is a certain zero.
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ long phase2Bytes = compressedBytesForRowRanges(blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = lateMatReader.getRowRanges(blockIdx);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private long compressedBytesForRowRanges(
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore =
+ wholeBlock ? null : lateMatReader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ // The store answers null, rather than throwing, for a path it was not built with. That can
+ // only mean the block's store was built while a narrower schema was requested than this
+ // walk asks about, an ordering bug rather than a property of the file, and the footer tells
+ // the two apart. It is reported rather than thrown: this walk only produces a counter, and
+ // `ignoreCorruptFiles` turns any exception from a reader into a silently truncated file, so
+ // a byte metric must not be able to change the answer.
+ if (chunk.getOffsetIndexReference() != null && !loggedMissingStoreEntry) {
+ loggedMissingStoreEntry = true;
+ LOG.warn("Undercounting the storage filter's avoided bytes for {}: column "
+ + path.toDotString() + " of row group " + blockIndex + " has an offset index the "
+ + "block's column index store was not built with",
+ MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ }
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below: the bytes buffered for
+ // splicing, and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = storageFilter.test(keyScratchBatch.getRow(r));
+ } catch (RuntimeException e) {
+ if (!storageFilter.isEvaluationError(e)) throw e;
+ // Fail open. The predicate ran on a row that, in the plan, an earlier conjunct would
+ // have rejected before it, so a plain scan never evaluates it there. Giving the filter up
+ // for this row group puts every row the pushed filter kept back in the output, and the
+ // post-scan Filter then evaluates the conjuncts in their own order. Either an earlier one
+ // drops the row before this expression runs, or it does not and the query fails the way
+ // it would have without this feature.
+ logFilterGivenUpOnError(e);
+ giveUpFilter();
+ return null;
+ }
+ if (survives) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) splicedBytes += appendSurvivorRowToAccumulators(r);
+ // Both halves of what this row group retains grow per survivor, and either can cross
+ // the budget on its own, so they are weighed together here and nowhere else. The cheaper
+ // concession comes first: release the buffer, and give the filter up as well if the
+ // ranges alone still do not fit.
+ long rangeBytes = rowRangeStateBytes(survivorRangeCount);
+ if (splicedBytes + rangeBytes > cap) {
Review Comment:
**The survivor buffer is outside any `MemoryConsumer`, and this budget undercounts it.** The check counts `1 + 8` bytes per variable-length key row plus the value length, and 40 bytes per range. It does not count:
- the capacity-sized accumulator allocation itself;
- the byte child, which starts at `capacity * 4` and doubles on growth;
- the child's `nulls` array, which is as large as its data;
- the next accumulator, which is allocated eagerly after each push.
For a ~10-byte string key and capacity 4096, one accumulator actually holds ~168KB while ~78KB is counted (2.16x). At the 64MB default that is ~145MB per task, e.g. ~4.6GB on a 32-core executor. None of it is visible to the `MemoryManager`, and with off-heap column vectors none of it counts against `spark.memory.offHeap.size` either, so it shows up as a container kill or an OOM that cannot spill. The conf doc says "not the backing arrays", but not the 2-4x magnitude. Could this go through a `MemoryConsumer`? `abandonSplicing()` already maps naturally to `spill()`. Alternatively, the budget could at least count the real vector capacities.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
Review Comment:
On a file without an offset index (pyarrow's default `write_page_index=False`, or pre-1.11 parquet-mr), the only useful thing phase 1 can learn is whether a row group has zero survivors. Still, it decodes and evaluates every key row and builds the full `RowRanges`, and then the row group is read in full, keys included.
On top of that, the missing index is learned only from the first phase-2 `MissingOffsetIndexException`. So in each split, the first partially-surviving row group is fully spliced (up to 64MB of keys copied) and then discarded.
Two cheap improvements:
- once `fileHasNoOffsetIndex` is known, stop `evaluateStorageFilter` at the first survivor;
- decide up front from the footer (`getOffsetIndexReference() == null` for the phase-2 columns), and keep the catch as the fallback for an index that is present but unreadable.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Asked this way rather than up front, from the footer. Parquet resolves the index over the
+ // paths current at its first lookup for the block, which without a pushed data filter is
+ // the non-key columns alone, so a footer walk over the projection is both stricter than the
+ // read and blind to an index that is claimed but unreadable. It is also where the throw
+ // costs least: it lands before any data page is read.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Reading {} without page-level storage filtering: reading part of a row group "
Review Comment:
A file without an offset index is an expected, documented degrade mode, yet this logs a WARN with a full stack trace (`e`) once per file, since `fileHasNoOffsetIndex` is per reader. On a table of thousands of pyarrow-written files, that floods the executor logs. parquet's own `ColumnIndexFilter` handles the same exception with a single INFO line and no stack trace.
Could this be INFO/DEBUG without the throwable, or logged once per executor? The same applies to `logFilterGivenUpOnError`, which is also suppressed only per file.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Asked this way rather than up front, from the footer. Parquet resolves the index over the
+ // paths current at its first lookup for the block, which without a pushed data filter is
+ // the non-key columns alone, so a footer walk over the projection is both stricter than the
+ // read and blind to an index that is claimed but unreadable. It is also where the throw
+ // costs least: it lands before any data page is read.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Reading {} without page-level storage filtering: reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs. Row groups the filter empties are still skipped whole", e,
+ MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ // The retry only avoids the same wall because a store missing one column's offset index
+ // reports no column index either, so these ranges cover the whole block and parquet reads
+ // it without consulting an index. That is three parquet internals deep, so it is checked:
+ // a release that changes any of them should fail here rather than throw from the read.
+ if (baselineRows != blockRowCount) {
+ throw new IllegalStateException(String.format(
+ "Cannot read row group %d of %s without an offset index: the pushed filter selects "
+ + "%d of %d rows, so a plain read of the block is not what it asks for",
+ blockIdx, lateMatReader.getFile(), baselineRows, blockRowCount));
+ }
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up: it read what a plain scan
+ // reads, so the saving is a certain zero.
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ long phase2Bytes = compressedBytesForRowRanges(blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = lateMatReader.getRowRanges(blockIdx);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private long compressedBytesForRowRanges(
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore =
+ wholeBlock ? null : lateMatReader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ // The store answers null, rather than throwing, for a path it was not built with. That can
+ // only mean the block's store was built while a narrower schema was requested than this
+ // walk asks about, an ordering bug rather than a property of the file, and the footer tells
+ // the two apart. It is reported rather than thrown: this walk only produces a counter, and
+ // `ignoreCorruptFiles` turns any exception from a reader into a silently truncated file, so
+ // a byte metric must not be able to change the answer.
+ if (chunk.getOffsetIndexReference() != null && !loggedMissingStoreEntry) {
+ loggedMissingStoreEntry = true;
+ LOG.warn("Undercounting the storage filter's avoided bytes for {}: column "
+ + path.toDotString() + " of row group " + blockIndex + " has an offset index the "
+ + "block's column index store was not built with",
+ MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ }
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below: the bytes buffered for
+ // splicing, and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = storageFilter.test(keyScratchBatch.getRow(r));
+ } catch (RuntimeException e) {
+ if (!storageFilter.isEvaluationError(e)) throw e;
+ // Fail open. The predicate ran on a row that, in the plan, an earlier conjunct would
+ // have rejected before it, so a plain scan never evaluates it there. Giving the filter up
+ // for this row group puts every row the pushed filter kept back in the output, and the
+ // post-scan Filter then evaluates the conjuncts in their own order. Either an earlier one
+ // drops the row before this expression runs, or it does not and the query fails the way
+ // it would have without this feature.
+ logFilterGivenUpOnError(e);
+ giveUpFilter();
+ return null;
+ }
+ if (survives) {
+ finalRangesBuilder.addSelectedRow(blockRow);
Review Comment:
There is no density check on the survivor ranges. With unclustered keys, survivors are scattered: at a 10-50% survival rate there are roughly f(1-f)·R ranges, up to ~250K for 1M rows. Every non-key page still overlaps a survivor, so no IO is saved. Meanwhile phase 2 decodes every non-key column range by range (`nextRange` + `skipValues` + a 1-value read in `VectorizedRleValuesReader.readBatchInternal`) instead of one bulk decode per page. That could make non-key decode CPU several times higher than a plain read.
`rowRangeStateBytes` bounds memory only (~1.6M ranges at 64MB). Would it make sense to `giveUpFilter()` when ranges/keptRows exceeds a threshold, or when every non-key page overlaps a survivor? A benchmark with random survivors at, say, 3%, 25% and 50% would show where the break-even is.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
Review Comment:
`FileSourceStrategy.storageFiltersFor` already returns `Nil` when every read data column is a key, so this all-keys mode (`nonKeyColumns == null`) is reachable only by driving the reader directly, as the comment at L806 says. Falling back with `storageFilter = null; return;` when `nonKeyFieldCount == 0` would remove:
- this branch;
- the `dataPages != null` guard;
- `columns == null` in `compressedBytesForRowRanges`.
Production behavior would not change. The catch is that the tests driving key-only reads (the key-type × dictionary matrix, etc.) would need a non-key column added. Without it, the tests that keep every row would pass silently without exercising the survivor queues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5776202009
@cloud-fan, @dongjoon-hyun, @sunchao, can you please take a look at this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4133526563
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -409,23 +631,250 @@ public boolean nextBatch() throws IOException {
}
cv.assemble();
}
- // If needed, compute row indexes within a file.
- if (rowIndexGenerator != null) {
- rowIndexGenerator.populateRowIndex(columnVectors, num);
- }
+ }
- rowsReturned += num;
+ /** Publishes {@code num} rows as the current batch. */
+ private void finishBatch(int num) {
columnarBatch.setNumRows(num);
+ rowsReturned += num;
numBatched = num;
batchIdx = 0;
- return true;
+ }
+
+ /**
+ * Points the batch's key slots at one survivor key vector each, which is what publishes them: the
+ * same {@link ColumnarBatch} is handed out every time, over an array it holds by reference. The
+ * queues keep owning those vectors until the next batch releases them, so a phase-2 read that
+ * throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void publishSurvivorKeyVectors(int num) {
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ if (keyVectorQueues[i].isEmpty()) {
+ // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop
+ // is driven by that same count. Named rather than left to NoSuchElementException.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to "
+ + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num));
+ }
+ }
+ keyVectorsPublished = true;
+ // Queue k holds the survivors of key column k, and `keyColumnIndices[k]` is the batch slot that
+ // key column sits in, so the pairing is read off rather than re-derived from slot order. Only
+ // key slots are touched, since a non-key slot never holds anything but its persistent vector.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int k = 0; k < keyIndices.length; k++) {
+ spliceBatchColumns[keyIndices[k]] = keyVectorQueues[k].peekFirst();
+ }
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (java.util.ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
missingColumns = new HashSet<>();
for (ParquetColumn column : CollectionConverters.asJava(parquetColumn.children())) {
checkColumn(column);
}
+ if (storageFilter != null) {
+ initializeLateMaterialization();
+ }
+ }
+
+ /**
+ * Sets the storage filter for late materialization. Must be called before {@link #initialize};
+ * {@link #initializeLateMaterialization()} then inspects the per-file schema and decides whether
+ * splicing engages.
+ *
+ * <p>It does not engage in one real case: every key column is missing from this physical file
+ * under schema evolution. The predicate is then rewritten with each missing key replaced by the
+ * constant the reader materializes for it (its existence DEFAULT, else null) and evaluated once.
+ * True keeps the file unfiltered, false or null skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by
+ * {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}, so a
+ * violation of it here is a planner bug and is asserted rather than handled. A file the reader
+ * simply cannot prune is a different matter: it reads it the way a plain scan would.
+ */
+ public void setStorageFilter(ParquetStorageFilter storageFilter) {
+ this.storageFilter = storageFilter;
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (!configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true)) {
Review Comment:
Fixed the way you describe. The conf no longer turns the feature off. It sets the same flag a missing index sets, and `pushedFilterRangesFor` returns `RowRanges.createSingle(blockRowCount)` for phase 0 and for the whole-file skip path, which is where `getRowRanges` ignoring `useColumnIndexFilter()` would otherwise have bitten. So the filter keeps the row groups it empties, which need no index at all. A test runs both arms and asserts the metrics, so it cannot pass by the feature being off in both.
One thing came out of that flag afterwards. It carried two meanings, the conf and a partial read that had failed, and for the second one dropping `getRowRanges` for the rest of the file dropped the pushed data filter's own page pruning too, which a plain read keeps. Measured at 215 rows against a plain read's 160, on a three-row-group file with one damaged index. There are two flags now, and the latch takes the narrowing alone.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
Review Comment:
Both done. There is a footer check per row group now, and phase 1 stops at the first survivor when the row group cannot be narrowed, since whether anything survives at all is the only thing left to learn there. Parquet's exception stays as the fallback for an index the footer claims and parquet cannot then produce.
One correction to the check as you describe it. Asking about the phase-2 columns alone is not enough, because parquet builds a block's `ColumnIndexStore` over the requested schema at the first lookup, and with a data filter pushed that is phase 0's, over the whole projection. So a key column with no offset index empties the store as well. The check asks about the whole projection for that reason, and a test on a hand-built file whose key column is the only unindexed chunk fails if it asks about the non-key columns alone.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Asked this way rather than up front, from the footer. Parquet resolves the index over the
+ // paths current at its first lookup for the block, which without a pushed data filter is
+ // the non-key columns alone, so a footer walk over the projection is both stricter than the
+ // read and blind to an index that is claimed but unreadable. It is also where the throw
+ // costs least: it lands before any data page is read.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Reading {} without page-level storage filtering: reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs. Row groups the filter empties are still skipped whole", e,
+ MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ // The retry only avoids the same wall because a store missing one column's offset index
+ // reports no column index either, so these ranges cover the whole block and parquet reads
+ // it without consulting an index. That is three parquet internals deep, so it is checked:
+ // a release that changes any of them should fail here rather than throw from the read.
+ if (baselineRows != blockRowCount) {
+ throw new IllegalStateException(String.format(
+ "Cannot read row group %d of %s without an offset index: the pushed filter selects "
+ + "%d of %d rows, so a plain read of the block is not what it asks for",
+ blockIdx, lateMatReader.getFile(), baselineRows, blockRowCount));
+ }
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up: it read what a plain scan
+ // reads, so the saving is a certain zero.
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ long phase2Bytes = compressedBytesForRowRanges(blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = lateMatReader.getRowRanges(blockIdx);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private long compressedBytesForRowRanges(
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore =
+ wholeBlock ? null : lateMatReader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ // The store answers null, rather than throwing, for a path it was not built with. That can
+ // only mean the block's store was built while a narrower schema was requested than this
+ // walk asks about, an ordering bug rather than a property of the file, and the footer tells
+ // the two apart. It is reported rather than thrown: this walk only produces a counter, and
+ // `ignoreCorruptFiles` turns any exception from a reader into a silently truncated file, so
+ // a byte metric must not be able to change the answer.
+ if (chunk.getOffsetIndexReference() != null && !loggedMissingStoreEntry) {
+ loggedMissingStoreEntry = true;
+ LOG.warn("Undercounting the storage filter's avoided bytes for {}: column "
+ + path.toDotString() + " of row group " + blockIndex + " has an offset index the "
+ + "block's column index store was not built with",
+ MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ }
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below: the bytes buffered for
+ // splicing, and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = storageFilter.test(keyScratchBatch.getRow(r));
+ } catch (RuntimeException e) {
+ if (!storageFilter.isEvaluationError(e)) throw e;
+ // Fail open. The predicate ran on a row that, in the plan, an earlier conjunct would
+ // have rejected before it, so a plain scan never evaluates it there. Giving the filter up
+ // for this row group puts every row the pushed filter kept back in the output, and the
+ // post-scan Filter then evaluates the conjuncts in their own order. Either an earlier one
+ // drops the row before this expression runs, or it does not and the query fails the way
+ // it would have without this feature.
+ logFilterGivenUpOnError(e);
+ giveUpFilter();
+ return null;
+ }
+ if (survives) {
+ finalRangesBuilder.addSelectedRow(blockRow);
Review Comment:
Measured, and at the extreme it is worse than the guess. On a synthetic table of one long key and twenty long payload columns written in 256-row pages, with one survivor every other row, phase 2 skips no page at all and the read takes 5.6x the plain one. It falls to 1.8x at runs of sixteen rows. So the shape you name is real, and it is the one a gate needs most. The numbers are in the description.
I did not add the gate, because it cannot be as simple as a ranges-per-row ratio. Giving up also re-reads the key columns and hands the post-scan `Filter` every row the filter would have dropped, and those costs grow with what the filter excluded rather than with what it kept, so the direction flips as the filter gets more selective. That wants a cost model, which is the second follow-up in the description and one of the reasons the conf is off by default.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -409,23 +631,250 @@ public boolean nextBatch() throws IOException {
}
cv.assemble();
}
- // If needed, compute row indexes within a file.
- if (rowIndexGenerator != null) {
- rowIndexGenerator.populateRowIndex(columnVectors, num);
- }
+ }
- rowsReturned += num;
+ /** Publishes {@code num} rows as the current batch. */
+ private void finishBatch(int num) {
columnarBatch.setNumRows(num);
+ rowsReturned += num;
numBatched = num;
batchIdx = 0;
- return true;
+ }
+
+ /**
+ * Points the batch's key slots at one survivor key vector each, which is what publishes them: the
+ * same {@link ColumnarBatch} is handed out every time, over an array it holds by reference. The
+ * queues keep owning those vectors until the next batch releases them, so a phase-2 read that
+ * throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void publishSurvivorKeyVectors(int num) {
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ if (keyVectorQueues[i].isEmpty()) {
+ // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop
+ // is driven by that same count. Named rather than left to NoSuchElementException.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to "
+ + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num));
+ }
+ }
+ keyVectorsPublished = true;
+ // Queue k holds the survivors of key column k, and `keyColumnIndices[k]` is the batch slot that
+ // key column sits in, so the pairing is read off rather than re-derived from slot order. Only
+ // key slots are touched, since a non-key slot never holds anything but its persistent vector.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int k = 0; k < keyIndices.length; k++) {
+ spliceBatchColumns[keyIndices[k]] = keyVectorQueues[k].peekFirst();
+ }
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (java.util.ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
missingColumns = new HashSet<>();
for (ParquetColumn column : CollectionConverters.asJava(parquetColumn.children())) {
checkColumn(column);
}
+ if (storageFilter != null) {
+ initializeLateMaterialization();
+ }
+ }
+
+ /**
+ * Sets the storage filter for late materialization. Must be called before {@link #initialize};
+ * {@link #initializeLateMaterialization()} then inspects the per-file schema and decides whether
+ * splicing engages.
+ *
+ * <p>It does not engage in one real case: every key column is missing from this physical file
+ * under schema evolution. The predicate is then rewritten with each missing key replaced by the
+ * constant the reader materializes for it (its existence DEFAULT, else null) and evaluated once.
+ * True keeps the file unfiltered, false or null skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by
+ * {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}, so a
+ * violation of it here is a planner bug and is asserted rather than handled. A file the reader
+ * simply cannot prune is a different matter: it reads it the way a plain scan would.
+ */
+ public void setStorageFilter(ParquetStorageFilter storageFilter) {
+ this.storageFilter = storageFilter;
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (!configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true)) {
+ // That conf is the escape hatch for a file whose page index is wrong, so it has to turn this
+ // feature off whole rather than only its filtering. Phase 2 reads part of a row group through
+ // the offset index, which parquet consults whatever the conf says, and a wrong one there
+ // pairs a row's key with another row's values. The post-scan filter cannot catch that: the
+ // key it sees is the right one.
+ storageFilter = null;
+ return;
+ }
+ lateMatReader = reader.getUnderlyingReader();
+ if (lateMatReader == null) {
+ // Late materialization drives a ParquetFileReader directly, so without one the filter is
+ // simply not applied and the plain read path runs. The conjunct is in the post-scan filter as
+ // well, so the rows it would have dropped are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ // Resolve each key column's top-level ParquetColumn. Partition into present and missing (the
+ // latter can happen under schema evolution: a column is in the requested schema but not in this
+ // physical parquet file). For any non-primitive key we still bail; phase-1 reads only primitive
+ // leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> missingKeyLocalPositions = new ArrayList<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (missingColumns.contains(column)) {
+ missingKeyLocalPositions.add(i);
+ } else {
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // If any key column is missing from this file, rewrite the predicate to substitute the constant
+ // the reader will actually materialize for that column. That is the column's existence DEFAULT
+ // when it has one (ParquetColumnVector writes it into the output vector and marks the vector
+ // constant), otherwise null. Substituting null for a column that reads back as its default
+ // would filter on a value the scan never returns. The predicate must be evaluated against the
+ // substituted constant rather than skipped: null does not always mean false in a filter.
+ if (!missingKeyLocalPositions.isEmpty()) {
+ int[] missing = new int[missingKeyLocalPositions.size()];
+ Object[] missingValues = new Object[missingKeyLocalPositions.size()];
+ Object[] existenceDefaults =
+ ResolveDefaultColumns.existenceDefaultValues(sparkRequestedSchema);
+ for (int i = 0; i < missing.length; i++) {
+ missing[i] = missingKeyLocalPositions.get(i);
+ missingValues[i] = existenceDefaults[keyIndices[missing[i]]];
+ }
+ storageFilter = storageFilter.rewriteForMissingKeys(missing, missingValues);
Review Comment:
Adopted, all three parts. The predicate and `keyColumnIndices` are unchanged, the row phase 1 evaluates takes the vector `initBatch` built for a key column this file does not have, and splicing covers only the keys the file does have. The rewrite, its per-task cache and the invariant it needed are gone.
The all-keys-missing decision moved into `initBatch`, after `super.initBatch`, which is where those vectors exist. So there is one rule for what a missing column reads as, and it is `ParquetColumnVector`'s.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Asked this way rather than up front, from the footer. Parquet resolves the index over the
+ // paths current at its first lookup for the block, which without a pushed data filter is
+ // the non-key columns alone, so a footer walk over the projection is both stricter than the
+ // read and blind to an index that is claimed but unreadable. It is also where the throw
+ // costs least: it lands before any data page is read.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Reading {} without page-level storage filtering: reading part of a row group "
Review Comment:
Fixed for that one. It logs the exception's message rather than the exception, since a file written without a page index is a shape this feature expects and degrades on. It stays `WARN` and stays once per file, because which files degrade is what someone tuning this needs, and the line is one line.
`logFilterGivenUpOnError` keeps the exception on purpose. That is the error the query would have raised without this feature, and it is the user's own expression that raised it, so the stack trace is the useful part of it. It is also once per file.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
Review Comment:
Done. The reader answers an all-keys projection with `storageFilter = null` and reads plainly, so that branch, the `dataPages` guard and the `columns == null` arm of the byte walk are all gone.
Your prediction about the tests was right. Every key-type fixture carries a non-key column now, and the helper that writes them says why, so a read that keeps every row still goes through the survivor queues.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.SparkThrowable
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Literal, Predicate, XxHash64}
+import org.apache.spark.sql.catalyst.trees.TreePattern.PLAN_EXPRESSION
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.{BinaryType, BooleanType, ByteType, DataType, DateType, DayTimeIntervalType, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType, StringType, StructType, TimestampNTZType, TimestampType, TimeType, YearMonthIntervalType}
+
+/**
+ * Optional SQL metrics the reader updates while applying a [[ParquetStorageFilter]]. Every
+ * counter is scoped to what the storage filter added on top of a read of the same projection
+ * without one. All fields are nullable; a null field disables that metric.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums the rows those skips excluded, per skipped block the rows that
+ * survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it, and it is zero on an
+ * all-keys projection, which can avoid nothing.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned.
+ *
+ * The row counters' suffix says where a row was excluded, not what would have saved it: a row
+ * inside a kept row group is read as part of its page and dropped during decode, so page
+ * filtering did not save it. An all-keys projection has no page filtering at all, and
+ * [[rowsExcludedWithinRowGroup]] still counts every row the filter dropped.
+ */
+case class StorageFilterMetrics(
Review Comment:
Done. `StorageFilterMetrics` takes all five as required fields, the null checks and the `needBytes` mode are gone, and the `storageFilterMetrics` parameter has no default any more.
The keys and the labels moved onto its companion, so no format code names one, and the tests build real `SQLMetric`s through the same `create` the production path uses.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -642,7 +656,26 @@ trait FileSourceScanLike extends DataSourceScanExec with SessionStateHelper {
} else {
None
}
- } ++ driverMetrics
+ } ++ storageFilterMetrics ++ driverMetrics
+
+ protected lazy val storageFilterMetrics: Map[String, SQLMetric] = if (storageFilters.nonEmpty) {
Review Comment:
Done, with the hook you propose. `FileFormat.storageFilterMetrics(sparkContext)` answers with the format's own metric map, and `FileSourceScanLike` adds whatever it gets to `metrics`. So the scan names nothing, and a format that counts something else declares it. What there is to count is the format's own vocabulary, since row groups and page filtering are Parquet's words.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5936075195
@dongjoon-hyun thank you for another careful round. Round 8 is answered in the threads, and the description is up to date.
Several of these found real problems, and I fixed them together rather than one by one:
- A fatal error now goes out as itself, found as deep as the executor would look had a plain plan raised it. A kill keeps its checked wrapper.
- Every non-fatal error from the filter falls back, an `AssertionError` included. So does a codec `InternalError` while decoding a key page, which then fails at the same batch as a plain read.
- A failure copying a survivor gives splicing up with a WARN of its own, and the per-type tests now fail on one. The reader's own invariants throw an internal error that `ignoreCorruptFiles` cannot take for a corrupt file.
- A later accumulator set is sized up front only when its charge fits under the cap.
- A `MissingOffsetIndexException` costs only its own row group.
One more blind review round of ours after these found a few more, fixed in the same push:
- An `InternalError` from the filter goes out in a checked wrapper, since `ignoreCorruptFiles` would take a bare one for a corrupt file.
- A later set is sized up front only if that fits after the survivor's own charge.
- Each kind of give-up WARN is reported once per split, rather than one for all of them.
- New tests pin the metrics and the budget exactly.
The branch is also rebased onto master. That brings SPARK-59828, without which a v2 file's `DELTA_BYTE_ARRAY` decimal column failed when survivors scatter.
There is also a new `StorageFilterPushdownBenchmark`, in its own commit, as you asked in round 4 (https://github.com/apache/spark/pull/58895#discussion_r4108708299). It runs a join whose runtime bloom filter is the storage filter, with the feature off and on, over clustered keys and over keys scattered at 3%, 25% and 50%. The results files come from the GitHub Actions benchmark workflow.
A few items repeat findings that earlier threads answered:
- the 2-arg `initialize` override, which has to stay for MiMa (https://github.com/apache/spark/pull/58895#discussion_r4135661837);
- the `ParquetReadState` shape, which the round-6 benchmark settled (https://github.com/apache/spark/pull/58895#discussion_r4137343688).
And one goes against an earlier ask of yours. The second subclass check is the one you asked for in round 4 (https://github.com/apache/spark/pull/58895#discussion_r4108708266), and it is one private predicate now.
Could the next round treat the settled ones as settled unless something new comes up? It would help keep the review on what is still open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4109912730
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -771,6 +836,28 @@ case class FileSourceScanExec(
readRDD
}
+ // Materialize scalar subqueries inside storage filters to literals and bind AttributeReferences
+ // to BoundReferences targeting positions in `requiredSchema`. Subqueries must have been prepared
+ // by SparkPlan before this is forced (same contract as `pushedDownFilters`).
+ @transient
+ protected lazy val preparedStorageFilters: Seq[Expression] = {
+ if (storageFilters.isEmpty) {
+ Nil
+ } else {
+ // No conf check here: the conf decides at planning time whether a scan is offered storage
+ // filters at all, and re-reading it now could only make this scan drop work it already has.
+ //
+ // `output` is `readDataColumns ++ generatedMetadataColumns ++ partitionColumns ++
+ // constantMetadataColumns` and `requiredSchema` is the StructType of the first two groups, so
+ // the first `requiredSchema.length` attributes line up with its fields.
+ val requestedDataAttrs = output.take(requiredSchema.length)
+ storageFilters.map { expr =>
+ val subqueryReplaced = expr.transform { case s: execution.ScalarSubquery => s.toLiteral }
+ BindReferences.bindReference(subqueryReplaced, requestedDataAttrs)
Review Comment:
**Non-blocking (P2):** `supportsStorageFilter` sees the original predicate, but this preparation replaces every `AttributeReference` with a `BoundReference` before `buildReaderWithStorageFilters` runs. That contradicts the documented same-expression handoff and means a format that opted in using a column name or field metadata cannot apply the same decision while constructing its reader. Could we carry both the original expression and an executable bound form (or otherwise preserve the advertised information)? Please also add an end-to-end generic `FileFormat` test that exercises the planner-to-reader handoff and the builder-returning-`None` fallback to `buildReaderWithPartitionValues`.
**Recommended change:** Represent the original offered expression and its executable prepared form explicitly at the FileFormat handoff, so the builder can observe the promised original information while Parquet still evaluates a materialized, bound form. Add generic FileFormat integration coverage for identity/metadata preservation and for None falling back to the ordinary reader.
**Why this works:** Prepare each accepted predicate without discarding its original representation, pass both representations through a single aligned handoff object or equivalent paired contract, and make Parquet consume the bound member. Exercise the full planning and scan-construction path with a test format that records the original expression and can decline specialized construction.
**Scope:** Refine the generic V1 storage-filter contract, its planner-to-scan transport, the Parquet consumer adaptation, and regression coverage.
**Compatibility:** Storage filtering remains a default-off optional optimization; unsupported formats and runtime give-up paths continue to return ordinary scan results.
**Risks:** The two representations could become misaligned if they are stored as independent sequences rather than one paired value. Changing the newly added FileFormat method shape must keep default decline behavior and Parquet serialization intact.
**Constraints:** Scalar subqueries must still be materialized before reader closure serialization. Parquet evaluation must remain bound to requiredSchema ordinals. The post-scan Filter remains authoritative and the optimization remains optional.
**Success:** A format can correlate the exact named/metadata-bearing expression it accepted with the executable predicate used to construct its reader. Parquet receives a materialized expression bound to the requested schema and preserves current results and fallback behavior. Returning None from the specialized builder invokes the ordinary builder and returns its rows. A regression that canonicalizes or discards the original expression before reader construction fails the new integration coverage.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1947,6 +1947,51 @@ object SQLConf {
.checkValue(threshold => threshold >= 0, "The threshold must not be negative.")
.createWithDefault(10)
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED =
+ buildConf("spark.sql.parquet.storageFilterPushdown.enabled")
+ .doc("If true, allows the vectorized Parquet reader to evaluate runtime storage filters " +
+ "(e.g. bloom filters from join runtime filtering) at the scan level using late " +
+ "materialization: read key columns first, evaluate the filter per row, then read data " +
+ "columns restricted to surviving rows. This is a planning-time decision only: when " +
+ "false, no storage filter is attached to a scan in the first place and the filter is " +
+ "applied as an ordinary post-scan filter alone. A pushed filter stays in the post-scan " +
+ "filter as well, the way a pushed data filter does, so honoring it is optional: a reader " +
+ "that meets a row group it cannot prune reads it the way a plain scan would, plus one " +
+ "more read of the key columns, since the phase that evaluated the filter already read " +
+ "them. A file written with no Parquet page index is read with the filter applied only " +
+ "where it empties a whole row group, since narrowing to part of one needs that index, so " +
+ "every row group of it that keeps a row pays that. Setting " +
+ "parquet.filter.columnindex.enabled to false turns this off entirely, because reading " +
+ "part of a row group goes through the page index. Note that " +
+ "the surviving key values of a whole row group are buffered before the " +
+ "first batch of that row group is produced, so a task holds up to one extra copy of the " +
+ "key columns for one row group.")
+ .version("5.0.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES =
+ buildConf("spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes")
+ .internal()
+ .doc("Most memory, in bytes, that the vectorized Parquet reader holds for one row group " +
Review Comment:
**Nit (P3):** The opening `Most memory ...` is a sentence fragment, and the following independent clauses are joined with commas. Since this text appears in the generated configuration reference, could you rewrite it as a complete definition (for example, `The maximum memory ...`) and split the accounting rules into complete sentences?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java:
##########
@@ -19,30 +19,29 @@
import org.apache.parquet.column.ColumnDescriptor;
-import java.util.ArrayList;
-import java.util.Iterator;
-import java.util.List;
import java.util.PrimitiveIterator;
/**
* Helper class to store intermediate state while reading a Parquet column chunk.
*/
final class ParquetReadState {
- /** A special row range used when there is no row indexes (hence all rows must be included) */
- private static final RowRange MAX_ROW_RANGE = new RowRange(Long.MIN_VALUE, Long.MAX_VALUE);
+ /** The row indexes to include, only not-null if the column index is present. */
Review Comment:
**Nit (P3):** Non-null row indexes are not conditioned solely on a column index. The late-materialization path can pass `finalRanges` to `readFilteredRowGroup`, which derives row indexes from offset indexes even when the optional column index is absent. Could this comment describe the actual condition under which `rowIndexes` is populated?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.SparkThrowable
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Literal, Predicate, XxHash64}
+import org.apache.spark.sql.catalyst.trees.TreePattern.PLAN_EXPRESSION
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.{BinaryType, BooleanType, ByteType, DataType, DateType, DayTimeIntervalType, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType, StringType, StructType, TimestampNTZType, TimestampType, TimeType, YearMonthIntervalType}
+
+/**
+ * Optional SQL metrics the reader updates while applying a [[ParquetStorageFilter]]. Every
+ * counter is scoped to what the storage filter added on top of a read of the same projection
+ * without one. All fields are nullable; a null field disables that metric.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums the rows those skips excluded, per skipped block the rows that
+ * survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it, and it is zero on an
+ * all-keys projection, which can avoid nothing.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned.
+ *
+ * The row counters' suffix says where a row was excluded, not what would have saved it: a row
+ * inside a kept row group is read as part of its page and dropped during decode, so page
Review Comment:
**Nit (P3):** `rowsExcludedWithinRowGroup` counts all rows rejected inside a retained row group, but it does not prove that every rejected row's page was read: phase 2 can skip a page containing no surviving rows. Could this describe only what the counter establishes, without the row/page IO claim? The adjacent `rowsExcludedByRowGroup` bullet also needs a complete sentence.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2526 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayOutputStream, File}
+import java.net.URI
+import java.time.LocalTime
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapreduce.Job
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName}
+import org.apache.parquet.hadoop.util.HadoopOutputFile
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, XxHash64}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FileSourceScanLike, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests the late-materialization path of [[VectorizedParquetRecordReader]] driven by a
+ * [[ParquetStorageFilter]]. Writes small multi-row-group parquet files, wires a hand-built filter
+ * into the reader, and asserts correctness + the two storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false), StructField("v", StringType, nullable = false)))
+ ParquetStorageFilter.create(Seq(expr), requested, metrics)
+ }
+
+ // Writes a single-column (just `k`) parquet file for the supplied key type via Spark's
+ // {@code Encoder}.
+ private def writeKeyOnlyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(dir, spark.createDataset(keys).toDF("k"), rowGroupSize)
+
+ // Reads a key-only file, returning the survivor keys and the reader. The {@code extract} function
+ // pulls one value at a time from the batch's key column.
+ private def readKeyOnlyAll[T](
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T,
+ capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k"), storageFilter, (batch, i) => extract(batch.column(0), i),
+ capacity)
+
+ // Builds a `k >= threshold` storage filter bound to position 0 against a key-only schema of the
+ // given key type.
+ private def keyOnlyAtLeastFilter(
+ threshold: Literal,
+ keyType: DataType,
+ metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, keyType, nullable = false), threshold)
+ val requested = StructType(Seq(StructField("k", keyType, nullable = false)))
+ ParquetStorageFilter.create(Seq(expr), requested, metrics)
+ }
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows in one row group: parquet's first row-group size check is at record 100, so
+ // `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = keyAtLeastFilter(1000L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf,
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(rgSkipped.value > 0,
+ s"expected at least one row group skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedRg.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${rowsExcludedRg.value}")
+ assert(rowsExcludedPf.value == 0,
+ s"no partial-row-group filtering expected; got ${rowsExcludedPf.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(bytesAvoidedRg.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got ${bytesAvoidedRg.value}")
+ assert(bytesAvoidedPf.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got ${bytesAvoidedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val filter = keyAtLeastFilter(0L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped, rowsExcludedWithinRowGroup = rowsExcludedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(rgSkipped.value == 0, s"nothing should be skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedPf.value == 0,
+ s"nothing should be filtered; got ${rowsExcludedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("mixed: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ // Output is exact: VectorizedColumnReader uses PageReadStore.getRowIndexes (driven by
+ // our finalRanges) to skip rows within partial pages, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it: a row group that was neither skipped whole nor emitted has
+ // to have its rows counted there.
+ assert(rowsExcludedPf.value > 0,
+ s"expected rows excluded inside a kept row group; got ${rowsExcludedPf.value}")
+ assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == rows.size,
+ s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " +
+ s"should account for all ${rows.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key-only projection: phase 2 is skipped and both byte-avoided metrics are zero") {
+ // When the projected schema contains only the bloom key, phase 2 is skipped entirely
+ // (`nonKeyColumns == null` in the reader). All output rows come from the per-key-column
+ // queues populated in phase 1. Total bytes read match the no-storage-filter path (phase 1 reads
+ // the key column once instead of phase 2 re-reading it), so both `avoided` metrics are zero:
+ // there are no non-key bytes to skip. This is the shape the design notes call the biggest win,
+ // so it must not be the shape that pays for metrics.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ // k >= 195 -> last 6 keys survive; preceding row groups skipped or page-pruned.
+ val filter = keyOnlyAtLeastFilter(Literal(195L), LongType, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf,
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i))
+ try {
+ val expected = keys.filter(_ >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact survivor keys; got ${result.sorted}, expected ${expected.toSeq.sorted}")
+ assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedRg.value > 0,
+ s"a skipped row group must count its rows too; got ${rowsExcludedRg.value}")
+ // The all-keys path takes its kept-row count from `finalRowCount` rather than from a
+ // phase-2 page store, so this identity is the only thing that checks that arithmetic.
+ assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == keys.size,
+ s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " +
+ s"should account for all ${keys.size} rows")
+ // For an all-keys projection, the no-storage-filter path would have read the same key
+ // column phase 1 reads. There are no non-key bytes to avoid; both metrics are 0.
+ assert(bytesAvoidedRg.value == 0,
+ s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedRg.value}")
+ assert(bytesAvoidedPf.value == 0,
+ s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("multi-batch emit: survivor count exceeds capacity") {
+ // Drive the reader at capacity = 16 with a row group of 100 surviving rows. Exercises:
+ // - The per-key-column queue holding multiple full-capacity vectors plus a partial tail.
+ // - The published queue head getting closed at the start of every subsequent emit.
+ // - The batch's key slots being rewritten ceil(100/16) = 7 times.
+ withTempDir { dir =>
+ val keys = (1L to 100L)
+ // Big rowGroupSize so all 100 rows fit in one row group.
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 64 * 1024L)
+ // Filter accepts every row so the queue is fully populated.
+ val filter = keyOnlyAtLeastFilter(Literal(0L), LongType)
+ val (result, reader) =
+ readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i), capacity = 16)
+ try {
+ assert(result == keys.toSeq,
+ s"expected all keys returned in order across multiple batches; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("int key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the IntegerType branch of ValueCopier.
+ withTempDir { dir =>
+ val keys = (1 to 100)
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyOnlyAtLeastFilter(Literal(90), IntegerType)
+ val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getInt(i))
+ try {
+ assert(result.toSet == keys.filter(_ >= 90).toSet,
+ s"expected int keys >= 90; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("string key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the StringType branch of ValueCopier (variable-length byte copy via putByteArray).
+ withTempDir { dir =>
+ val keys = (1 to 20).map(i => f"k$i%03d")
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyOnlyAtLeastFilter(
+ Literal.create("k015", StringType), StringType)
+ val (result, reader) =
+ readKeyOnlyAll(path, filter, (vec, i) => vec.getUTF8String(i).toString)
+ try {
+ assert(result.toSet == keys.filter(_ >= "k015").toSet,
+ s"expected string keys >= 'k015'; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction: storageFiltersFor pre-checks each one, and
+ // by the time create runs the conjunct is gone from the post-scan Filter, so a soft rejection
Review Comment:
**Nit (P3):** Two comments in this suite describe a different contract from the one being tested: the suite exercises five storage-filter metrics rather than two, and `FileSourceStrategy` deliberately retains this conjunct in the post-scan `Filter` while also attaching it to the scan. Could you update both comments so they document the actual coverage and fallback-safety mechanism?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4084862663
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its "
+ + "filter kept in row group %d of %s, but the file was written without a page "
+ + "index. Set %s=false to read this file.",
+ finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(),
+ SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new UnsupportedFileReadException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it --
+ // which can make the row group's contribution negative, and that is the truth about it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
Review Comment:
This evaluation now runs inside the file iterator, i.e. inside `FileScanRDD`'s `ignoreCorruptFiles` catch. `shouldIgnoreCorruptFileException` excludes only `UnsupportedFileReadException`, so with `spark.sql.files.ignoreCorruptFiles=true` any evaluation error is treated as a corrupt file. An ANSI cast error from the key expression (see the comment on `FileSourceStrategy`) is enough. The rest of the file, including the not-yet-emitted survivors of the current row group, is then silently skipped and the query succeeds. Before this PR the same exception came from `FilterExec`, outside that catch, and failed the query. The existing `ignoreCorruptFiles` test covers only `UnsupportedFileReadException`.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -151,6 +151,60 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
}
}
+ /**
+ * Splits `afterScanFilters` into conjuncts the file format can evaluate at the storage layer for
+ * late materialization (returned as the first element) and the remaining filters that stay as a
+ * post-scan FilterExec (returned as the second element).
+ *
+ * Everything is checked here, statically, so the runtime never silently loses a filter. Two of
+ * the conditions are per scan, and failing either leaves every filter in the plan:
+ * - The storage-filter pushdown SQL conf is on.
+ * - [[FileFormat.supportBatch]] holds for the schema the reader will see,
+ * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives
+ * its own vectorized-read decision from. Late materialization needs a batch read, so this asks
+ * about batch support rather than naming a format.
+ *
+ * The rest are per conjunct, and one that fails any of them stays in the post-scan Filter:
+ * - It is deterministic. A reader evaluates the predicate without
+ * `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a
+ * `Nondeterministic` expression, so a non-deterministic conjunct would fail at task time.
+ * - It references at least one column, and every column it references is a projected data
+ * column. A reference to something the scan does not read cannot be evaluated by the reader.
+ * - [[FileFormat.supportsStorageFilter]] accepts it. That is where the expression shapes and
+ * column types a reader can evaluate live, so this method names neither a format nor a type.
+ *
+ * One last condition is on the set that survives: at least one projected data column must be left
+ * for the reader to prune. A scan that projects nothing but the filter's own key columns reads
+ * the same columns for the same rows either way, since the reader has to read a key column to
+ * evaluate the filter on it, so pushing can only add the cost of evaluating the predicate outside
+ * the generated code. Extraction is dropped entirely in that case.
+ */
+ private def extractStorageFilters(
+ afterScanFilters: ExpressionSet,
+ fsRelation: HadoopFsRelation,
+ readDataColumns: Seq[Attribute],
+ outputDataSchema: StructType): (Seq[Expression], ExpressionSet) = {
+ val sparkSession = fsRelation.sparkSession
+ val sqlConf = sparkSession.sessionState.conf
+ if (!sqlConf.parquetStorageFilterPushdownEnabled) return (Nil, afterScanFilters)
+ val resultSchema = StructType(fsRelation.partitionSchema.fields ++ outputDataSchema.fields)
+ if (!fsRelation.fileFormat.supportBatch(sparkSession, resultSchema)) {
+ return (Nil, afterScanFilters)
+ }
+
+ val dataAttrs = AttributeSet(readDataColumns)
+ val (eligible, rest) = afterScanFilters.partition { expr =>
Review Comment:
Extraction only checks `deterministic`, which changes the evaluation order. Before this PR, `CombineFilters` put the existing conjuncts first and the bloom was evaluated in `FilterExec` only on rows that passed them (short-circuit). Now phase 1 evaluates the bloom's value expression on every row in `pushedFilterRanges`.
Example: ANSI mode (the 4.x default), a string=bigint join key that `InjectRuntimeFilter` turns into `might_contain(bf, xxhash64(CAST(s AS BIGINT)))`, plus a guard like `WHERE kind = 'num'` over rows whose `s` is not numeric. This succeeds with the conf off and fails with `CAST_INVALID_INPUT` with the conf on. `evalAllMissing()` has the same issue, since it evaluates the expression against the DEFAULT value.
Adding a `throwable` check alone won't catch this, because `Cast` doesn't override `throwable`.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
Review Comment:
`getColumnIndexStore` memoizes one store per block, built at first call with the paths current then. Here that is init (`getFilteredRecordCount`) or phase 0, i.e. all requested columns. `ColumnIndexStoreImpl.create` returns `EMPTY` if any single column's offset index is missing, or if reading it throws an `IOException` (logged and treated as missing).
So when only a key column has the problem, this phase-2 read of non-key columns, which do have valid offset indexes, fails with "the file was written without a page index". The message is misleading, and `ignoreCorruptFiles` cannot skip the exception. The same file reads fine with no pushed data filter, or with the feature off.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its "
+ + "filter kept in row group %d of %s, but the file was written without a page "
+ + "index. Set %s=false to read this file.",
+ finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(),
+ SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new UnsupportedFileReadException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it --
+ // which can make the row group's contribution negative, and that is the truth about it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
Review Comment:
`finalRanges` is built one row at a time, so scattered survivors produce roughly N*p*(1-p) ranges. In phase 2, every non-key leaf's `VectorizedColumnReader` builds a `ParquetReadState`, which walks `getRowIndexes()` and keeps its own `List<RowRange>` until the next row group. For a 1M-row row group with p=0.1..0.5 and 40 non-key leaves, that is about 140-380 MB per task, and `maxSplicedRowGroupBytes` doesn't bound it. Before this PR, row ranges came only from the column index at page granularity, so there were few of them.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:
##########
@@ -230,6 +274,40 @@ class ParquetFileFormat
val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead
val archiveFormatEnabled = parquetOptions.archiveFormatEnabled
+ // Extraction has already removed these conjuncts from the post-scan Filter, so nothing else in
+ // the plan would apply them: anything that stops the reader from honoring them fails loudly
+ // rather than dropping them. `enableVectorizedReader` is one such thing, and it is recomputed
+ // from the live session conf when the RDD is built, so a flip of
+ // spark.sql.parquet.enableVectorizedReader (or the nested-column variant) after planning lands
+ // here.
+ val storageFilterOpt: Option[ParquetStorageFilter] = if (storageFilters.isEmpty) {
+ None
+ } else if (!enableVectorizedReader) {
+ throw new UnsupportedFileReadException(
Review Comment:
Here `storageFilters` are the prepared ones, so the bloom is a binary `Literal`. `Literal.toString` renders it as full hex, so with the default `maxNumBits` (8 MB bloom) this message can be about 16M characters, and it ends up in driver logs, the UI and the client. The same applies to the default `require` message in `FileFormat.buildReaderWithStorageFilters`. Could we truncate it or print only the key expressions?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its "
+ + "filter kept in row group %d of %s, but the file was written without a page "
+ + "index. Set %s=false to read this file.",
+ finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(),
+ SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new UnsupportedFileReadException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it --
+ // which can make the row group's contribution negative, and that is the truth about it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
+ }
+ }
+ }
+ remaining -= num;
+ }
+
+ if (accumulate) {
+ finalizePartialAccumulators();
+ }
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ // Assigned before the loop on purpose: an allocation failure part way through then leaves the
+ // vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows
+ * each.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, pushing them onto their
+ * queues once full. All key columns advance in lockstep, which is what keeps the queues aligned.
+ *
+ * <p>Returns false when the buffered survivors have passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} and this row group has
+ * given splicing up, in which case it has already released what it held and the caller must stop
+ * calling this.
+ */
+ private boolean appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ long valueBytes = 0L;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ // Measured on the destination: a dictionary-encoded source has no length of its own, since
+ // its values are read through the dictionary.
+ if (keyVariableLength[i]) valueBytes += dst.getArrayLength(dstRow);
+ }
+ }
+ splicedBytes += keyFixedBytesPerRow + valueBytes;
Review Comment:
The survivor buffer is outside any `MemoryConsumer`, and `splicedBytes` counts payload only. For variable-length keys, the byte child's doubling can make the real footprint up to about 2x the counted bytes, and `UnifiedMemoryManager` never sees it. With many concurrent tasks near the cap, co-located aggregates or sorts can OOM or thrash instead of spilling. Would it make sense to account this through a `MemoryConsumer` whose `spill()` calls `abandonSplicing()`?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -409,23 +598,276 @@ public boolean nextBatch() throws IOException {
}
cv.assemble();
}
- // If needed, compute row indexes within a file.
- if (rowIndexGenerator != null) {
- rowIndexGenerator.populateRowIndex(columnVectors, num);
- }
+ }
- rowsReturned += num;
+ /** Publishes {@code num} rows as the current batch. */
+ private void finishBatch(int num) {
columnarBatch.setNumRows(num);
+ rowsReturned += num;
numBatched = num;
batchIdx = 0;
+ }
+
+ /**
+ * Splicing emit path. Publishes one survivor key vector per key column into the batch, reads the
+ * non-key slots for {@code num} rows, and hands out the same {@link ColumnarBatch} every time,
+ * its key slots rewritten in place. The batch is a view over vectors owned elsewhere; see
+ * {@link #close()} and {@link #closeSplicingState()}.
+ */
+ private boolean nextBatchSplicing() throws IOException {
+ releasePublishedKeyVectors();
+ for (ParquetColumnVector cv : columnVectors) {
+ cv.reset();
+ }
+ // Zero the outgoing batch before the terminal checks below, as the plain path does. Otherwise a
+ // terminal call leaves the batch pointing at key vectors that were just closed -- off-heap,
+ // that is freed memory.
+ if (columnarBatch != null) columnarBatch.setNumRows(0);
+ if (hitEndOfData) return false;
+ if (rowsReturned >= totalRowCount) return false;
+ checkEndOfRowGroup();
+ if (hitEndOfData) return false;
+
+ int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned);
+
+ if (!spliceCurrentRowGroup) {
+ // This row group was read the plain way: phase 2 took every projected column, so every slot
+ // of the batch comes from the persistent one.
+ readPersistentColumns(num, /* skipKeySlots= */ false);
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.populateRowIndex(columnVectors, num);
+ }
+ System.arraycopy(persistentBatchColumns, 0, spliceBatchColumns, 0, spliceBatchColumns.length);
+ finishBatch(num);
+ return true;
+ }
+
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ if (keyVectorQueues[i].isEmpty()) {
+ // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop
+ // is driven by that same count. Named rather than left to NoSuchElementException because
+ // this one must not be swallowed: under ignoreCorruptFiles that would silently drop the
+ // rest of a healthy file's rows.
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to "
+ + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num));
+ }
+ }
+ // The queues keep owning these until the next emit releases them, so a phase-2 read below that
+ // throws leaves them reachable for `close()`.
+ keyVectorsPublished = true;
+ // Key slots are filled in ascending slot order while `keyIdx` walks the queues in key-list
+ // order, so the pairing is the identity only because `ParquetStorageFilter.create` sorts
+ // `keyColumnIndices` ascending. `isKeyTopLevel` says which slots are keys, not where each
+ // sits in that list, so this loop cannot re-derive the pairing: an unsorted list would swap
+ // key columns in the output batch.
+ int keyIdx = 0;
+ for (int i = 0; i < spliceBatchColumns.length; i++) {
+ if (i < isKeyTopLevel.length && isKeyTopLevel[i]) {
+ spliceBatchColumns[i] = keyVectorQueues[keyIdx++].peekFirst();
+ } else {
+ spliceBatchColumns[i] = persistentBatchColumns[i];
+ }
+ }
+
+ readPersistentColumns(num, /* skipKeySlots= */ true);
+ if (rowIndexGenerator != null) {
+ // Row-index column is identified by name (ROW_INDEX_TEMPORARY_COLUMN_NAME), which is a
+ // synthetic metadata column never referenced by a storage filter, so its slot is a non-key
+ // slot with a persistent ParquetColumnVector.
+ rowIndexGenerator.populateRowIndex(columnVectors, num);
+ }
+ finishBatch(num);
return true;
}
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (java.util.ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
private void initializeInternal() throws IOException, UnsupportedOperationException {
missingColumns = new HashSet<>();
for (ParquetColumn column : CollectionConverters.asJava(parquetColumn.children())) {
checkColumn(column);
}
+ if (storageFilter != null) {
+ initializeLateMaterialization();
+ }
+ }
+
+ /**
+ * Sets the storage filter for late materialization. Must be called before {@link #initialize};
+ * {@link #initializeLateMaterialization()} then inspects the per-file schema and decides whether
+ * splicing engages.
+ *
+ * <p>It does not engage in one real case: every key column is missing from this physical file
+ * under schema evolution. The predicate is then rewritten with each missing key replaced by the
+ * constant the reader materializes for it (its existence DEFAULT, else null) and evaluated once
+ * -- true keeps the file unfiltered, false or null skips it.
+ *
+ * <p>Every other precondition is guaranteed by
+ * {@code FileSourceStrategy.extractStorageFilters} and {@code ParquetStorageFilter.create}, and a
+ * violation throws rather than falling back: extraction has already removed the filter from the
+ * post-scan Filter, so not applying it would return rows the query rejected.
+ */
+ public void setStorageFilter(ParquetStorageFilter storageFilter) {
+ this.storageFilter = storageFilter;
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ lateMatReader = reader.getUnderlyingReader();
+ if (lateMatReader == null) {
+ // Unreachable: the only ParquetRowGroupReader ParquetFileFormat builds is
+ // ParquetRowGroupReaderImpl, which exposes its reader.
+ throw new UnsupportedFileReadException(
+ "Storage-filter pushdown requires a reader backed by a ParquetFileReader, but "
+ + reader.getClass().getName() + " does not expose one");
+ }
+ if (configuration != null) {
+ useColumnIndexFilter = configuration.getBoolean(
+ ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ }
+
+ // Resolve each key column's top-level ParquetColumn. Partition into present and missing (the
+ // latter can happen under schema evolution: a column is in the requested schema but not in this
+ // physical parquet file). For any non-primitive key we still bail; phase-1 reads only primitive
+ // leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> missingKeyLocalPositions = new ArrayList<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new UnsupportedFileReadException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (missingColumns.contains(column)) {
+ missingKeyLocalPositions.add(i);
+ } else {
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // If any key column is missing from this file, rewrite the predicate to substitute the constant
+ // the reader will actually materialize for that column. That is the column's existence DEFAULT
+ // when it has one (ParquetColumnVector writes it into the output vector and marks the vector
+ // constant), otherwise null. Substituting null for a column that reads back as its default
+ // would filter on a value the scan never returns. The predicate must be evaluated against the
+ // substituted constant rather than skipped: null does not always mean false in a filter.
+ if (!missingKeyLocalPositions.isEmpty()) {
+ int[] missing = new int[missingKeyLocalPositions.size()];
+ Object[] missingValues = new Object[missingKeyLocalPositions.size()];
+ Object[] existenceDefaults =
+ ResolveDefaultColumns.existenceDefaultValues(sparkRequestedSchema);
+ for (int i = 0; i < missing.length; i++) {
+ missing[i] = missingKeyLocalPositions.get(i);
+ missingValues[i] = existenceDefaults[keyIndices[missing[i]]];
+ }
+ storageFilter = storageFilter.rewriteForMissingKeys(missing, missingValues);
+
+ if (presentKeyColumns.isEmpty()) {
Review Comment:
When every key column is missing and the rewritten predicate is false, the whole file is skipped without updating `rowGroupsSkipped` / `rowsExcludedByRowGroup` / `bytesAvoidedByRowGroup`. That doesn't match their documented meaning.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its "
+ + "filter kept in row group %d of %s, but the file was written without a page "
+ + "index. Set %s=false to read this file.",
+ finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(),
+ SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new UnsupportedFileReadException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it --
+ // which can make the row group's contribution negative, and that is the truth about it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
+ }
+ }
+ }
+ remaining -= num;
+ }
+
+ if (accumulate) {
+ finalizePartialAccumulators();
+ }
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ // Assigned before the loop on purpose: an allocation failure part way through then leaves the
+ // vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows
+ * each.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, pushing them onto their
+ * queues once full. All key columns advance in lockstep, which is what keeps the queues aligned.
+ *
+ * <p>Returns false when the buffered survivors have passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} and this row group has
+ * given splicing up, in which case it has already released what it held and the caller must stop
+ * calling this.
+ */
+ private boolean appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ long valueBytes = 0L;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ // Measured on the destination: a dictionary-encoded source has no length of its own, since
+ // its values are read through the dictionary.
+ if (keyVariableLength[i]) valueBytes += dst.getArrayLength(dstRow);
+ }
+ }
+ splicedBytes += keyFixedBytesPerRow + valueBytes;
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ // Only here, so the cost is one comparison per capacity-sized vector rather than per row. A
+ // row group whose survivors fit in a single accumulator is never checked at all: it then
+ // holds one capacity-sized vector per key column, which is what a plain read holds anyway.
+ if (splicedBytes > storageFilter.maxSplicedRowGroupBytes()) {
+ abandonSplicing();
+ return false;
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+ }
+ return true;
+ }
+
+ /**
+ * Gives up splicing for the row group being evaluated and releases every survivor vector it has
+ * buffered. Phase 2 then reads the full projected schema and the emit path takes the persistent
+ * batch, so the rows are unaffected.
+ */
+ private void abandonSplicing() {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ closeAll(currentKeyAccumulators);
+ Arrays.fill(currentKeyAccumulators, null);
+ currentKeyAccumulatorRowCount = 0;
+ spliceCurrentRowGroup = false;
+ }
+
+ /**
+ * Pushes any partially-filled accumulator into its queue at row-group end so the emit path can
+ * dequeue it as the row group's final batch.
+ */
+ private void finalizePartialAccumulators() {
+ if (currentKeyAccumulatorRowCount == 0) return;
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Closes anything held by the splicing path: every vector still queued, the published head
+ * included, and partially-filled accumulators. Called from {@link #close()}.
+ */
+ private void closeSplicingState() {
+ keyVectorsPublished = false;
+ if (keyVectorQueues != null) {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ if (q != null) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ }
+ }
+ closeAll(currentKeyAccumulators);
+ currentKeyAccumulators = null;
+ }
+
+ /** Closes every non-null vector of {@code vectors}; tolerates a null array. */
+ private static void closeAll(WritableColumnVector[] vectors) {
+ if (vectors == null) return;
+ for (WritableColumnVector v : vectors) {
+ if (v != null) v.close();
+ }
+ }
+
+ /**
+ * Copies one key value between column vectors. Picked per key column at init time by
+ * {@link #copierFor(DataType)}; the caller handles null sources.
+ */
+ @FunctionalInterface
+ private interface ValueCopier {
+ void copy(WritableColumnVector dst, int dstRow, WritableColumnVector src, int srcRow);
+ }
+
+ /**
+ * Returns a {@link ValueCopier} for the given key {@link DataType}. The set of types handled here
+ * is the definition behind {@code ParquetStorageFilter.isSupportedKeyType}, which gates both
+ * planning-time extraction and {@code ParquetStorageFilter.create} -- so the throw at the end is
+ * unreachable. Teach both sides at once when adding a type; a type admitted there but missing
+ * here becomes a task failure instead of a planning-time rejection.
+ */
+ private static ValueCopier copierFor(DataType dt) {
Review Comment:
`copierFor` re-implements the per-type row-to-vector copy that `RowToColumnConverter` (Columnar.scala) already provides, and it adds another hand-maintained type list that has to stay in lockstep with `isSupportedKeyType`, `isVariableLength` and the test's `renderValue`. If put-based copying is kept for performance, dispatching on `PhysicalDataType` / `DecimalType.is32BitDecimalType` etc., like `ParquetVectorUpdaterFactory` does, would at least share the mapping.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its "
+ + "filter kept in row group %d of %s, but the file was written without a page "
+ + "index. Set %s=false to read this file.",
+ finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(),
+ SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new UnsupportedFileReadException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it --
+ // which can make the row group's contribution negative, and that is the truth about it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
Review Comment:
`SQLMetric.add` ignores `v < 0` (`if (v >= 0)` in SQLMetrics.scala). The negative contribution the comment above describes ("that is the truth about it") is therefore dropped silently, and `bytesAvoidedByPageFiltering` over-reports for row groups that gave up splicing.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
Review Comment:
As far as I can see, this branch is unreachable in production. `FileSourceStrategy.extractStorageFilters` rejects all-keys projections, and `ParquetReadSupport.clipParquetGroupFields` keeps missing fields via `convertField`, so `nonKeyFieldCount` is always > 0, even under schema evolution. That contradicts the PR description ("It stays reachable under schema evolution") and the class comment that justifies splicing by this shape. Only the reader-level harness tests reach it. Should we drop the special case, or assert on it?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala:
##########
@@ -269,7 +269,15 @@ object DataSourceUtils extends PredicateHelper {
QueryExecutionErrors.sparkUpgradeInWritingDatesError(format, config)
}
+ /**
+ * Whether `ignoreCorruptFiles` may swallow this failure and skip the rest of its file.
+ *
+ * [[UnsupportedFileReadException]] is excluded because it does not report a corrupt file: it says
+ * the file cannot support a read the plan depends on. Skipping the rest of such a file would drop
+ * rows that are perfectly readable, and would do it silently.
+ */
def shouldIgnoreCorruptFileException(e: Throwable): Boolean = e match {
+ case _: UnsupportedFileReadException => false
Review Comment:
Design question: `InjectRuntimeFilter` only injects the bloom on a side the join already prunes (`canPruneLeft`/`canPruneRight`), so failing to apply it never changes the result. Could the extracted runtime filter be treated as optional instead? If the reader cannot honor it (no page index, vectorized conf flipped after planning, etc.), it would abandon splicing and read `pushedFilterRanges` plainly, the way DPP falls back to `TrueLiteral`. That would remove the new exception type, this special case in the shared classifier (also used by DSv2), and the new hard failure on files written without a page index (for example pyarrow's default). The trade-off is a documented contract that extracted filters must be result-preserving hints.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1885,6 +1885,42 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED =
Review Comment:
Nit: these confs (and their accessors near `parquetFilterPushDown`) split the `spark.sql.parquet.filterPushdown` group from its dependent sub-confs (`.date`, `.timestamp`, `.decimal`, `.string.startsWith`, `.stringPredicate`, `inFilterThreshold`). Storage-filter pushdown doesn't depend on `filterPushdown`, so could we move them after `PARQUET_FILTER_PUSHDOWN_INFILTERTHRESHOLD`?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Literal, Predicate}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.{BinaryType, BooleanType, ByteType, DataType, DateType, DayTimeIntervalType, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType, StringType, StructType, TimestampNTZType, TimestampType, TimeType, YearMonthIntervalType}
+
+/**
+ * Optional SQL metrics the reader updates while applying a [[ParquetStorageFilter]]. Every
+ * counter is scoped to what the storage filter added on top of a read of the same projection
+ * without one. All fields are nullable; a null field disables that metric.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums the rows those skips excluded, per skipped block the rows that
+ * survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it, and it is zero on an
+ * all-keys projection, which can avoid nothing.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned.
+ *
+ * The row counters' suffix says where a row was excluded, not what would have saved it: a row
+ * inside a kept row group is read as part of its page and dropped during decode, so page
+ * filtering did not save it. An all-keys projection has no page filtering at all, and
+ * [[rowsExcludedWithinRowGroup]] still counts every row the filter dropped.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric = null,
+ rowsExcludedByRowGroup: SQLMetric = null,
+ rowsExcludedWithinRowGroup: SQLMetric = null,
+ bytesAvoidedByRowGroup: SQLMetric = null,
+ bytesAvoidedByPageFiltering: SQLMetric = null)
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[boundExpression]] has its references rewritten to
+ * [[BoundReference]]s pointing at positions 0..(keyColumnIndices.length - 1); the reader must
+ * evaluate it against rows whose fields correspond to those key columns in that order.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ val boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer
+ // construction to first use on the executor.
+ @transient private lazy val predicate: BasePredicate = Predicate.create(boundExpression)
+
+ def test(keyRow: InternalRow): Boolean = predicate.eval(keyRow)
+
+ /**
+ * Returns a new filter for a physical file that is missing some key columns (schema evolution).
+ * The [[BoundReference]]s at `missingKeyLocalPositions`, which are indices into
+ * [[keyColumnIndices]], are replaced by `missingKeyValues`, and the remaining references are
+ * renumbered onto the reduced key-row layout. [[keyColumnIndices]] keeps the present columns in
+ * their original relative order, and SQL metrics are shared with `this`.
+ *
+ * `missingKeyValues(i)` must be the internal-format value the reader produces for a missing
+ * column: its existence DEFAULT when it has one, else null. `ParquetColumnVector` writes that
+ * default into the output vector, so substituting null instead would filter on a value the scan
+ * never returns and could drop matching rows.
+ *
+ * The predicate has to be evaluated against the substitution rather than skipped, because a null
+ * key does not always mean `false`: a `Coalesce`-wrapped reference still produces a non-null
+ * result, and `XxHash64` is `nullable = false` and hashes a null input to its seed.
+ *
+ * With every key position missing, the result holds no [[BoundReference]] at all and
+ * [[evalAllMissing]] can read off its constant truth value.
+ */
+ def rewriteForMissingKeys(
+ missingKeyLocalPositions: Array[Int],
+ missingKeyValues: Array[Any]): ParquetStorageFilter = {
+ require(missingKeyLocalPositions.length == missingKeyValues.length,
+ "missingKeyLocalPositions and missingKeyValues must have the same length")
+ val substitution = missingKeyLocalPositions.zip(missingKeyValues).toMap
+ val presentPositions = keyColumnIndices.indices.filterNot(substitution.contains)
+ val newPosOf = presentPositions.zipWithIndex.toMap
+ val rewritten = boundExpression.transform {
Review Comment:
`transform` rebuilds `BloomFilterMightContain` via `copy`, which drops its `@transient lazy val bloomFilter`. For every file with a missing key column, the whole bloom (up to about 8 MB) is therefore deserialized again, and `Predicate.create` runs again. The substituted values come from the scan schema and are the same for every file, so memoizing per missing-key set within a task would avoid this.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala:
##########
@@ -165,6 +166,57 @@ trait FileFormat {
}
}
+ /**
+ * Like [[buildReaderWithPartitionValues]] but additionally accepts a sequence of storage filters:
+ * Catalyst expressions that the storage layer may evaluate to drive value-column IO pruning based
+ * on key-column evaluation (e.g., late materialization with a runtime bloom filter).
+ *
+ * The default implementation delegates to [[buildReaderWithPartitionValues]] and accepts no
+ * storage filter at all. A format that supports storage-filter pushdown overrides this, and must
+ * not let the two builders call each other: this default delegates one way, so an override that
+ * delegates back recurses until the driver's stack runs out, `super` included, since that call is
+ * virtual too. Route both to a private implementation instead, the way `ParquetFileFormat` does.
+ *
+ * A non-empty `storageFilters` here is a planner bug, so the default body rejects it rather than
+ * dropping it: the planner removes an extracted conjunct from the post-scan `Filter`, so a reader
+ * that ignores it returns rows the filter rejects.
+ *
+ * Scalar subqueries inside `storageFilters` are expected to have been materialized before this
+ * method is called, so that the returned reader can be safely serialized to executors.
+ *
+ * `storageFilterMetrics` is an optional map of SQL metrics the reader can update during execution
+ * (e.g. number of row groups skipped). The scan is expected to expose these metrics via its
+ * `metrics` field so they show up in the SQL UI.
+ */
+ def buildReaderWithStorageFilters(
+ sparkSession: SparkSession,
+ dataSchema: StructType,
+ partitionSchema: StructType,
+ requiredSchema: StructType,
+ filters: Seq[Filter],
+ storageFilters: Seq[Expression],
+ options: Map[String, String],
+ hadoopConf: Configuration,
+ storageFilterMetrics: Map[String, SQLMetric] = Map.empty
+ ): PartitionedFile => Iterator[InternalRow] = {
+ require(storageFilters.isEmpty,
Review Comment:
The only caller, `FileSourceScanExec.inputRDD`, calls this only with non-empty `storageFilters`, which this `require` rejects. So the delegation to `buildReaderWithPartitionValues` below is unreachable. Yet it is exactly what creates the mutual-recursion trap the scaladoc warns third-party formats about. A default body that always throws, or a separate capability mixin with no default, would remove the trap.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +938,535 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes
+ // every projected column under finalRanges, key columns included, so nothing is buffered and
+ // the cost is one extra read of the key columns.
+ spliceCurrentRowGroup = true;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off -- the documented escape hatch for files whose column index is wrong.
+ // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of
+ // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is
+ // unaffected: it selects pages through the offset index, which this conf says nothing
+ // about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new UnsupportedFileReadException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ long finalRowCount = finalRanges.rowCount();
+
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely, which avoids
+ // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is
+ // not part of the baseline, so nothing has to be subtracted from it here.
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from
+ // the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ // An all-keys projection has nothing for phase 2 to read -- but only if the key values were
+ // buffered. A row group past the cap has to read them here like any other column.
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`),
+ // have none. Degrading to a whole-block read is not an option, because the key vectors
+ // hold only the survivors and the batch would misalign, and neither is dropping the
+ // predicate, which extraction has removed from the post-scan Filter. So the read fails,
+ // and all this adds is what the user can do about it.
+ //
+ // Nothing is checked up front: a row group the filter keeps whole never needs the index
+ // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and
+ // one it rejects whole is never read at all, so a file with no page index still scans as
+ // long as the filter never has to prune inside a row group.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ throw new UnsupportedFileReadException(String.format(
+ "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its "
+ + "filter kept in row group %d of %s, but the file was written without a page "
+ + "index. Set %s=false to read this file.",
+ finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(),
+ SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new UnsupportedFileReadException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it --
+ // which can make the row group's contribution negative, and that is the truth about it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
+ }
+ }
+ }
+ remaining -= num;
+ }
+
+ if (accumulate) {
+ finalizePartialAccumulators();
+ }
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ // Assigned before the loop on purpose: an allocation failure part way through then leaves the
+ // vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows
+ * each.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, pushing them onto their
+ * queues once full. All key columns advance in lockstep, which is what keeps the queues aligned.
+ *
+ * <p>Returns false when the buffered survivors have passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} and this row group has
+ * given splicing up, in which case it has already released what it held and the caller must stop
+ * calling this.
+ */
+ private boolean appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ long valueBytes = 0L;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ // Measured on the destination: a dictionary-encoded source has no length of its own, since
+ // its values are read through the dictionary.
+ if (keyVariableLength[i]) valueBytes += dst.getArrayLength(dstRow);
+ }
+ }
+ splicedBytes += keyFixedBytesPerRow + valueBytes;
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ // Only here, so the cost is one comparison per capacity-sized vector rather than per row. A
+ // row group whose survivors fit in a single accumulator is never checked at all: it then
+ // holds one capacity-sized vector per key column, which is what a plain read holds anyway.
+ if (splicedBytes > storageFilter.maxSplicedRowGroupBytes()) {
+ abandonSplicing();
+ return false;
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+ }
+ return true;
+ }
+
+ /**
+ * Gives up splicing for the row group being evaluated and releases every survivor vector it has
+ * buffered. Phase 2 then reads the full projected schema and the emit path takes the persistent
+ * batch, so the rows are unaffected.
+ */
+ private void abandonSplicing() {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ closeAll(currentKeyAccumulators);
+ Arrays.fill(currentKeyAccumulators, null);
+ currentKeyAccumulatorRowCount = 0;
+ spliceCurrentRowGroup = false;
+ }
+
+ /**
+ * Pushes any partially-filled accumulator into its queue at row-group end so the emit path can
+ * dequeue it as the row group's final batch.
+ */
+ private void finalizePartialAccumulators() {
+ if (currentKeyAccumulatorRowCount == 0) return;
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Closes anything held by the splicing path: every vector still queued, the published head
+ * included, and partially-filled accumulators. Called from {@link #close()}.
+ */
+ private void closeSplicingState() {
+ keyVectorsPublished = false;
+ if (keyVectorQueues != null) {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ if (q != null) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ }
+ }
+ closeAll(currentKeyAccumulators);
+ currentKeyAccumulators = null;
+ }
+
+ /** Closes every non-null vector of {@code vectors}; tolerates a null array. */
+ private static void closeAll(WritableColumnVector[] vectors) {
+ if (vectors == null) return;
+ for (WritableColumnVector v : vectors) {
+ if (v != null) v.close();
+ }
+ }
+
+ /**
+ * Copies one key value between column vectors. Picked per key column at init time by
+ * {@link #copierFor(DataType)}; the caller handles null sources.
+ */
+ @FunctionalInterface
+ private interface ValueCopier {
+ void copy(WritableColumnVector dst, int dstRow, WritableColumnVector src, int srcRow);
+ }
+
+ /**
+ * Returns a {@link ValueCopier} for the given key {@link DataType}. The set of types handled here
+ * is the definition behind {@code ParquetStorageFilter.isSupportedKeyType}, which gates both
+ * planning-time extraction and {@code ParquetStorageFilter.create} -- so the throw at the end is
+ * unreachable. Teach both sides at once when adding a type; a type admitted there but missing
+ * here becomes a task failure instead of a planning-time rejection.
+ */
+ private static ValueCopier copierFor(DataType dt) {
+ if (dt instanceof BooleanType) {
+ return (dst, dRow, src, sRow) -> dst.putBoolean(dRow, src.getBoolean(sRow));
+ }
+ if (dt instanceof ByteType) {
+ return (dst, dRow, src, sRow) -> dst.putByte(dRow, src.getByte(sRow));
+ }
+ if (dt instanceof ShortType) {
+ return (dst, dRow, src, sRow) -> dst.putShort(dRow, src.getShort(sRow));
+ }
+ if (dt instanceof IntegerType
+ || dt instanceof DateType
+ || dt instanceof YearMonthIntervalType) {
+ return (dst, dRow, src, sRow) -> dst.putInt(dRow, src.getInt(sRow));
+ }
+ if (dt instanceof LongType
+ || dt instanceof TimestampType
+ || dt instanceof TimestampNTZType
+ || dt instanceof TimeType
+ || dt instanceof DayTimeIntervalType) {
+ return (dst, dRow, src, sRow) -> dst.putLong(dRow, src.getLong(sRow));
+ }
+ if (dt instanceof FloatType) {
+ return (dst, dRow, src, sRow) -> dst.putFloat(dRow, src.getFloat(sRow));
+ }
+ if (dt instanceof DoubleType) {
+ return (dst, dRow, src, sRow) -> dst.putDouble(dRow, src.getDouble(sRow));
+ }
+ if (dt instanceof DecimalType decimalType) {
+ int precision = decimalType.precision();
+ if (precision <= Decimal.MAX_INT_DIGITS()) {
+ return (dst, dRow, src, sRow) -> dst.putInt(dRow, src.getInt(sRow));
+ }
+ if (precision <= Decimal.MAX_LONG_DIGITS()) {
+ return (dst, dRow, src, sRow) -> dst.putLong(dRow, src.getLong(sRow));
+ }
+ return (dst, dRow, src, sRow) -> dst.putByteArray(dRow, src.getBinary(sRow));
+ }
+ // StringType covers CHAR and VARCHAR: both extend it.
+ if (dt instanceof StringType || dt instanceof BinaryType) {
+ return (dst, dRow, src, sRow) -> dst.putByteArray(dRow, src.getBinary(sRow));
Review Comment:
On non-dictionary pages, `src.getBinary(sRow)` allocates a temporary `byte[]` for every surviving row, and the bytes are then copied a second time. High-cardinality string keys (UUIDs, for example) fall back to plain encoding, so this can mean millions of short-lived arrays per task. Could we copy directly from the source's byte storage by offset/length? The accumulators are also freshly allocated for every capacity-sized chunk and never pooled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4174361129
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/StorageFilterPushdownBenchmark.scala:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import java.io.File
+
+import org.apache.spark.SparkConf
+import org.apache.spark.benchmark.Benchmark
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.BloomFilterMightContain
+import org.apache.spark.sql.execution.{FileSourceScanExec, SparkPlan}
+import org.apache.spark.sql.execution.datasources.parquet.StorageFilterMetrics
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Benchmark to measure storage-filter pushdown in the vectorized Parquet reader. Every case runs
+ * the same join, whose runtime bloom filter is the storage filter, with
+ * `spark.sql.parquet.storageFilterPushdown.enabled` off and on.
+ *
+ * The fact table is written in its join key's order, so which keys a dimension table selects
+ * decides where the surviving rows fall:
+ * - clustered keys put the survivors in a few pages, so the reader skips the other row groups
+ * whole and the other pages of the row group that holds them. These cases show what the
+ * feature saves. The bloom filter's false positives land anywhere, though, so the more keys
+ * it holds, the more pages hold a survivor;
+ * - scattered keys put survivors in every page, so the reader can skip nothing and decodes the
+ * non-key columns range by range. These cases show what the feature costs.
+ * After each case it prints what the storage filter excluded, which does not depend on the
+ * machine.
+ *
+ * To run this benchmark:
+ * {{{
+ * 1. without sbt: bin/spark-submit --class <this class>
+ * --jars <spark core test jar>,<spark catalyst test jar> <spark sql test jar>
+ * 2. build/sbt "sql/Test/runMain <this class>"
+ * 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/Test/runMain <this class>"
+ * Results will be written to "benchmarks/StorageFilterPushdownBenchmark-results.txt".
+ * }}}
+ */
+object StorageFilterPushdownBenchmark extends SqlBasedBenchmark {
+
+ override def getSparkSession: SparkSession = {
+ val conf = new SparkConf()
+ .setAppName(this.getClass.getSimpleName)
+ // Since `spark.master` always exists, overrides this value
+ .set("spark.master", "local[1]")
+ .setIfMissing("spark.driver.memory", "3g")
+ .setIfMissing("spark.executor.memory", "3g")
+ .set(SQLConf.SHUFFLE_PARTITIONS.key, "1")
+ .set("spark.ui.enabled", "false")
+
+ SparkSession.builder().config(conf).getOrCreate()
+ }
+
+ private val numRows = 4L * 1024 * 1024
+ private val payloadColumns = 8
+
+ // The clustered cases select a contiguous range of keys from the middle of the table.
+ private val clustered = Seq("0.1%" -> 0.001, "1%" -> 0.01, "10%" -> 0.1)
+ // The scattered cases select keys by their hash, so the survivors are spread over every page.
+ private val scattered = Seq("3%" -> 3, "25%" -> 25, "50%" -> 50)
+
+ private def prepareTables(dir: File): Unit = {
+ val factPath = new File(dir, "fact").getCanonicalPath
+ // The payload is hashed, so it neither compresses nor dictionary-encodes away, and the bytes a
+ // skipped page saves are real.
+ val payload = (0 until payloadColumns).map(i => s"xxhash64(id, $i) AS v$i")
+ spark.range(0, numRows, 1, 1).selectExpr("id AS k" +: payload: _*).write.parquet(factPath)
+ spark.read.parquet(factPath).createOrReplaceTempView("fact")
+
+ def writeDimension(name: String, predicate: String): Unit = {
+ val path = new File(dir, name).getCanonicalPath
+ // `sel_id` exists for the query's filter on the dimension side, which is what makes the
+ // optimizer build a bloom filter from it at all.
+ spark.range(0, numRows, 1, 1).where(predicate).selectExpr("id AS k", "id AS sel_id")
+ .write.parquet(path)
+ spark.read.parquet(path).createOrReplaceTempView(name)
+ }
+ clustered.foreach { case (label, fraction) =>
+ val start = numRows / 2
+ val end = start + (numRows * fraction).toLong
+ writeDimension(dimension("clustered", label), s"id >= $start AND id < $end")
+ }
+ scattered.foreach { case (label, percent) =>
+ writeDimension(dimension("scattered", label), s"pmod(xxhash64(id), 100) < $percent")
+ }
+ }
+
+ private def dimension(layout: String, label: String): String =
+ s"${layout}_${label.replace("%", "").replace(".", "_")}"
+
+ private def query(dimension: String): String =
+ s"SELECT f.* FROM fact f JOIN $dimension d ON f.k = d.k WHERE d.sel_id >= 0"
+
+ private def factScan(plan: SparkPlan): FileSourceScanExec =
+ plan.collectFirst {
+ case scan: FileSourceScanExec if scan.output.exists(_.name == "v0") => scan
+ }.getOrElse(throw new IllegalStateException(s"No fact table scan in $plan"))
+
+ private def withPushdown[T](enabled: Boolean)(f: => T): T =
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> enabled.toString)(f)
+
+ // Runs the query once, untimed, and returns the fact scan with its metrics filled in.
+ private def executedFactScan(sql: String, enabled: Boolean): FileSourceScanExec =
+ withPushdown(enabled) {
+ val df = spark.sql(sql)
+ val scan = factScan(df.queryExecution.executedPlan)
+ val hasBloom = df.queryExecution.executedPlan.exists { node =>
+ node.expressions.exists(_.exists(_.isInstanceOf[BloomFilterMightContain]))
+ }
+ assert(hasBloom, s"The join must have a runtime bloom filter:\n${df.queryExecution}")
+ assert(scan.storageFilters.nonEmpty == enabled,
+ s"The fact scan must have a storage filter exactly when pushdown is on:\n$scan")
+ df.queryExecution.toRdd.foreach(_ => ())
+ scan
+ }
+
+ private def runCase(title: String, dimension: String): Unit = {
+ val sql = query(dimension)
+ val benchmark = new Benchmark(title, numRows, minNumIters = 3, output = output)
+ Seq(false, true).foreach { enabled =>
+ benchmark.addCase(s"Storage filter pushdown ${if (enabled) "on" else "off"}") { _ =>
+ withPushdown(enabled)(spark.sql(sql).noop())
+ }
+ }
+ benchmark.run()
+
+ val off = executedFactScan(sql, enabled = false)
Review Comment:
Done. The off arm only checks its plan now, and the line prints the table's row count for it.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1051 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
Review Comment:
Both done. The comments say it holds with `reserve`'s default doubling, and name the `hugeVectorReserveRatio` case. The fixed charge reads `1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW`.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while evaluating a storage filter, if it must not fall back
+ * on it by giving the filter up. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped, see [[throwIfKilled]]. The error may be the kill's
+ * interrupt surfacing through a UDF that wrapped it. `PythonRunner` reads a kill off the task
+ * context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had the post-scan Filter raised `e`, which it raises as it is, goes out
+ * as itself, so the executor finds it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in a checked wrapper. `FileScanRDD` would take a bare
+ * one for a corrupt file under `ignoreCorruptFiles`, where a plain plan raises it from the
+ * post-scan Filter, outside that catch. The executor finds it under the wrapper and the
+ * FAILED_READ_FILE around that at a `spark.executor.killOnFatalError.depth` of 3 or more, the
+ * default being 5.
+ *
+ * Anything else is the filter's own error, which the post-scan Filter raises again for a row it
+ * does evaluate, so the reader falls back on it.
+ */
+ def rethrowIfMustPropagate(e: Throwable): Unit = rethrowIfMustPropagate(e, fromRead = false)
+
+ /**
+ * As [[rethrowIfMustPropagate]], for an error decoding the key pages, with three differences:
+ * - a plain read raises that from the reader, where `FileScanRDD` wraps it in FAILED_READ_FILE
+ * before the executor looks, so a fatal error deeper in the chain has to sit one level
+ * shallower to count.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ * The reader falls back on those, and the plain read of the same pages meets the corrupt data
+ * again, or allocates what a plain read allocates.
+ */
+ def rethrowIfMustPropagateFromRead(e: Throwable): Unit =
+ rethrowIfMustPropagate(e, fromRead = true)
+
+ private def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new SparkException("A storage filter met a fatal error it must not absorb", internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ if (Executor.isFatalError(e, if (fromRead) fatalErrorDepth - 1 else fatalErrorDepth)) {
+ // The one `Executor.isFatalError` found, which no SparkOutOfMemoryError can precede.
+ val fatal = Iterator.iterate(e)(_.getCause).find(Utils.isFatalError).get
+ if (!corrupt(fatal)) throw out(fatal)
+ }
+ }
+
+ /**
+ * Whether `e` is how `WritableColumnVector.reserve` reports a vector that cannot grow: a plain
+ * `RuntimeException`, caused by an `OutOfMemoryError` or by nothing, for a capacity past what a
+ * vector can hold.
+ */
+ private def isVectorGrowthFailure(e: Throwable): Boolean =
+ e.getClass == classOf[RuntimeException] &&
+ (e.getCause == null || e.getCause.isInstanceOf[OutOfMemoryError])
+
+ /**
+ * Throws a pending task kill, for the reader's loops that can run long without returning to
+ * `FileScanRDD`, which is where a plain read meets one.
+ */
+ def throwIfKilled(): Unit = throwIfKilled(null)
+
+ /**
+ * Throws a pending task kill, wrapped in a checked [[SparkException]] around `surfaced`, the
+ * error the kill surfaced as, if any. Under `ignoreCorruptFiles`, `FileScanRDD` would log any
+ * `RuntimeException` or `IOException` from a reader as a corrupt file and go on to open the
+ * split's remaining files, since it checks for a kill only between batches. The executor reports
+ * the task as killed either way.
+ */
+ private def throwIfKilled(surfaced: Throwable): Unit = {
Review Comment:
Agreed, it is shared with a plain read. I added it to the description's follow-ups, with SPARK-50483 and SPARK-50545 as the precedent. The scaladoc here now says it covers the kills this reader sees at its own checks and catches, and names the two reads that leave a kill to `FileScanRDD` as a plain read does.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while evaluating a storage filter, if it must not fall back
+ * on it by giving the filter up. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped, see [[throwIfKilled]]. The error may be the kill's
+ * interrupt surfacing through a UDF that wrapped it. `PythonRunner` reads a kill off the task
+ * context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had the post-scan Filter raised `e`, which it raises as it is, goes out
+ * as itself, so the executor finds it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in a checked wrapper. `FileScanRDD` would take a bare
+ * one for a corrupt file under `ignoreCorruptFiles`, where a plain plan raises it from the
+ * post-scan Filter, outside that catch. The executor finds it under the wrapper and the
+ * FAILED_READ_FILE around that at a `spark.executor.killOnFatalError.depth` of 3 or more, the
+ * default being 5.
+ *
+ * Anything else is the filter's own error, which the post-scan Filter raises again for a row it
+ * does evaluate, so the reader falls back on it.
+ */
+ def rethrowIfMustPropagate(e: Throwable): Unit = rethrowIfMustPropagate(e, fromRead = false)
+
+ /**
+ * As [[rethrowIfMustPropagate]], for an error decoding the key pages, with three differences:
+ * - a plain read raises that from the reader, where `FileScanRDD` wraps it in FAILED_READ_FILE
+ * before the executor looks, so a fatal error deeper in the chain has to sit one level
+ * shallower to count.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ * The reader falls back on those, and the plain read of the same pages meets the corrupt data
+ * again, or allocates what a plain read allocates.
+ */
+ def rethrowIfMustPropagateFromRead(e: Throwable): Unit =
+ rethrowIfMustPropagate(e, fromRead = true)
+
+ private def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new SparkException("A storage filter met a fatal error it must not absorb", internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ if (Executor.isFatalError(e, if (fromRead) fatalErrorDepth - 1 else fatalErrorDepth)) {
+ // The one `Executor.isFatalError` found, which no SparkOutOfMemoryError can precede.
+ val fatal = Iterator.iterate(e)(_.getCause).find(Utils.isFatalError).get
+ if (!corrupt(fatal)) throw out(fatal)
+ }
+ }
+
+ /**
+ * Whether `e` is how `WritableColumnVector.reserve` reports a vector that cannot grow: a plain
+ * `RuntimeException`, caused by an `OutOfMemoryError` or by nothing, for a capacity past what a
+ * vector can hold.
+ */
+ private def isVectorGrowthFailure(e: Throwable): Boolean =
+ e.getClass == classOf[RuntimeException] &&
+ (e.getCause == null || e.getCause.isInstanceOf[OutOfMemoryError])
+
+ /**
+ * Throws a pending task kill, for the reader's loops that can run long without returning to
+ * `FileScanRDD`, which is where a plain read meets one.
+ */
+ def throwIfKilled(): Unit = throwIfKilled(null)
+
+ /**
+ * Throws a pending task kill, wrapped in a checked [[SparkException]] around `surfaced`, the
+ * error the kill surfaced as, if any. Under `ignoreCorruptFiles`, `FileScanRDD` would log any
+ * `RuntimeException` or `IOException` from a reader as a corrupt file and go on to open the
+ * split's remaining files, since it checks for a kill only between batches. The executor reports
+ * the task as killed either way.
+ */
+ private def throwIfKilled(surfaced: Throwable): Unit = {
+ val context = TaskContext.get()
+ if (context != null && context.isInterrupted()) {
+ val cause =
+ if (surfaced != null) surfaced else new TaskKilledException(context.getKillReason().get)
+ throw new SparkException("A storage filter met a task kill it must not absorb", cause)
+ }
+ }
+
+ /**
+ * Throws `SparkException.internalError(message, cause)`, for a broken invariant of this reader.
+ * The exception is checked, so `FileScanRDD` cannot read it as a corrupt file under
+ * `ignoreCorruptFiles` and skip the rest of the file silently. Typed as returning a
+ * `RuntimeException` only so that Java code can write `throw internalError(...)` without
+ * declaring a checked exception. It never returns.
+ */
+ def internalError(message: String, cause: Throwable): RuntimeException =
Review Comment:
Done, there is one overload now.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1051 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages {@link #reportOnce} has already reported. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /** Whether the reader has reported an offset index parquet could not read. */
+ private boolean reportedUnreadableOffsetIndex;
Review Comment:
Done. The message is a constant beside the others, gated by `reportedMessages`, and the boolean is gone. The damaged-index test now also damages two row groups and asserts one WARN.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether a log line containing `text` was logged.
+ private def withLogged[T](text: String)(body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains(text))
+ (result.get, logged)
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) =
+ withLogged("without page-level storage filtering")(body)
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def readWith(key: Expression): SparkException = intercept[SparkException] {
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ readAll(path, filter)._2.close()
+ }
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException = {
+ TaskContext.setTaskContext(TaskContext.empty())
+ try readWith(key) finally TaskContext.unset()
+ }
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_))),
+ classOf[SparkException])
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ val key = OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped")))
+ readAll(path, createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested))._2
+ .close()
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in a checked wrapper, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ assertPropagates(readWith(OnKey(bound, at = 7L, error = new InternalError(_))),
+ classOf[InternalError])
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val assertingKey = OnKey(bound, at = 7L, error = new AssertionError(_))
+ val (result, reader) =
+ readAll(path, createFilter(Seq(GreaterThanOrEqual(assertingKey, Literal(10L))), requested))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagate(under(depth - 1, oom))
+ } eq oom, "the deepest level the executor sees counts")
+ ParquetStorageFilter.rethrowIfMustPropagate(under(depth, oom))
+
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(depth - 2, oom))
+ } eq oom, "a decoding error's chain counts one level less deep")
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(depth - 1, oom))
+
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(1, new InternalError("codec")))
+ // One from an expression goes out in a checked wrapper, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file, and the executor finds it under it.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[SparkException](ParquetStorageFilter.rethrowIfMustPropagate(e))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ assert(Executor.isFatalError(thrown, depth), s"got $thrown")
+ }
+ ParquetStorageFilter.rethrowIfMustPropagate(new AssertionError("not fatal"))
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM.
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(growth)
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagate(growth)
+ }.getMessage == "reserve")
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.setTaskContext(context)
+ try {
+ assert(intercept[OutOfMemoryError](ParquetStorageFilter.rethrowIfMustPropagate(oom)) eq oom)
+ val killed = intercept[SparkException] {
+ ParquetStorageFilter.rethrowIfMustPropagate(under(1, oom))
+ }
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ } finally {
+ TaskContext.unset()
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ TaskContext.setTaskContext(TaskContext.empty())
+ val thrown = try {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ } finally {
+ TaskContext.unset()
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty)
+ .getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ val (_, sevenUncapped, _) = run("64MB", 394L)
+ val (_, sevenAtCharge, _) = run("103b", 394L)
+ val (_, sevenUnderCharge, _) = run("102b", 394L)
+ assert(sevenAtCharge == sevenUncapped,
+ s"a cap of exactly the charge must keep splicing; got $sevenAtCharge against " +
+ sevenUncapped)
+ assert(sevenUnderCharge > sevenUncapped,
+ s"a cap one byte under it must give splicing up; got $sevenUnderCharge against " +
+ sevenUncapped)
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a later accumulator set is sized up front only when its charge fits under the cap") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, which takes that memory before any value arrives, so it is charged then. Six survivors
+ // of a 1000-byte key over a capacity of 4 fill one set of four, charged about 16KB, and sizing
+ // the next set would charge as much again for two survivors. Under a cap of 28000 bytes that
+ // does not fit, so the set grows value by value instead, and the row group stays spliced, which
+ // reading the key column only once shows. The sizing has to fit after the fifth survivor's own
+ // fixed 17 bytes too. A cap of 32120 leaves 15995 for it, so the set grows and stays spliced.
+ // Under a cap that fits it, the second set is sized and charged once. A cap of 32130 holds
+ // the 32125 bytes of the fifth survivor and gives splicing up at the sixth, at 32142. A cap
+ // of 40000 keeps splicing. A missing or a double charge would each get one of these wrong.
+ // And the keys that land in the second set must come back right either way.
+ withTempDir { dir =>
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), REPEAT('x', 996)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val fileSchema = kvSchema(keyType = StringType)
+ val storageFilters = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, fileSchema, storageFilters, maxSplicedBytes, batchSize = 4)
+
+ val (uncappedRows, uncappedBytes) = run("64MB")
+ val (cappedRows, cappedBytes) = run("28000b")
+ assert(uncappedRows == 6 && cappedRows == 6,
+ s"the filter keeps keys 15..20 either way; got $uncappedRows and $cappedRows")
+ assert(cappedBytes == uncappedBytes,
+ "a set whose sizing does not fit must grow instead, so the row group stays spliced and " +
+ s"reads the key column once; capped=$cappedBytes uncapped=$uncappedBytes")
+ val (_, noRoomBytes) = run("32120b")
+ assert(noRoomBytes == uncappedBytes,
+ "a sizing that fits only without this survivor's own charge must not happen, so a cap of " +
+ s"32120 bytes keeps splicing; got $noRoomBytes against $uncappedBytes")
+ val (_, justUnderBytes) = run("32130b")
+ assert(justUnderBytes > uncappedBytes,
+ "the sizing must be charged when the set is sized, which passes a cap of 32130 bytes; " +
+ s"got $justUnderBytes against $uncappedBytes")
+ val (_, roomyBytes) = run("40000b")
+ assert(roomyBytes == uncappedBytes,
+ "and charged once, so a cap of 40000 bytes keeps splicing; " +
+ s"got $roomyBytes against $uncappedBytes")
+
+ val expected = (15 to 20).map(id => (f"$id%04d" + "x" * 996, id.toString))
+ Seq(Long.MaxValue, 28000L).foreach { cap =>
+ val (pairs, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(storageFilters, fileSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString),
+ capacity = 4)
+ try {
+ assert(pairs == expected,
+ s"every survivor must come back with its own key under cap $cap; got " +
+ pairs.map(_._2))
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ val outDir = new File(dir, "damaged").getAbsolutePath
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ rows.toDF("k", "v").repartition(1).write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 64L * 1024 * 1024)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .option(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT, 100L)
+ .parquet(outDir)
+ val path = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ .head.getAbsolutePath
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(false, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(false, 16, storageFilter)
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ (read.toSeq, error)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError) = readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ }
+ }
+
+ test("a key value the filter does not read and cannot be decoded gives splicing up") {
+ // The filter need not read every key on every row, as `coalesce(a, b)` reads `b` only where
+ // `a` is null. The copy into the survivor buffer reads every key, so it can meet a value no
+ // plain read decodes. A dictionary-encoded column decodes a value only when it is read, so a
+ // dictionary that holds fewer values than its ids point at is such a value. Copying it gives
+ // splicing up for the row group, and phase 2 reads the key columns the way a plain read does.
+ // So the read answers where a plain one does, and the filter still applies.
+ withTempDir { dir =>
+ // Two copies of each `b`, so parquet keeps its dictionary. Cut to 64 values, it leaves `b`
+ // undecodable from row 128 on. `a` is null only before that, so the filter never reads a
+ // damaged `b`.
+ val rows = (0 until 200).map { i =>
+ (if (i < 100 && i % 5 == 0) null else s"a_$i", s"b_${i / 2}", i)
+ }
+ val outDir = new File(dir, "dictionary").getAbsolutePath
Review Comment:
Done. `writeSingleParquetFile` takes an `options` map, applied last. The three new tests and the corrupt-key-page test go through it, and the v2 test uses `ParquetOutputFormat.WRITER_VERSION`.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether a log line containing `text` was logged.
+ private def withLogged[T](text: String)(body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains(text))
+ (result.get, logged)
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) =
+ withLogged("without page-level storage filtering")(body)
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def readWith(key: Expression): SparkException = intercept[SparkException] {
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ readAll(path, filter)._2.close()
+ }
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException = {
+ TaskContext.setTaskContext(TaskContext.empty())
+ try readWith(key) finally TaskContext.unset()
+ }
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_))),
+ classOf[SparkException])
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ val key = OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped")))
+ readAll(path, createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested))._2
+ .close()
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in a checked wrapper, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ assertPropagates(readWith(OnKey(bound, at = 7L, error = new InternalError(_))),
+ classOf[InternalError])
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val assertingKey = OnKey(bound, at = 7L, error = new AssertionError(_))
+ val (result, reader) =
+ readAll(path, createFilter(Seq(GreaterThanOrEqual(assertingKey, Literal(10L))), requested))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagate(under(depth - 1, oom))
+ } eq oom, "the deepest level the executor sees counts")
+ ParquetStorageFilter.rethrowIfMustPropagate(under(depth, oom))
+
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(depth - 2, oom))
+ } eq oom, "a decoding error's chain counts one level less deep")
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(depth - 1, oom))
+
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(1, new InternalError("codec")))
+ // One from an expression goes out in a checked wrapper, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file, and the executor finds it under it.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[SparkException](ParquetStorageFilter.rethrowIfMustPropagate(e))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ assert(Executor.isFatalError(thrown, depth), s"got $thrown")
+ }
+ ParquetStorageFilter.rethrowIfMustPropagate(new AssertionError("not fatal"))
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM.
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(growth)
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagate(growth)
+ }.getMessage == "reserve")
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.setTaskContext(context)
+ try {
+ assert(intercept[OutOfMemoryError](ParquetStorageFilter.rethrowIfMustPropagate(oom)) eq oom)
+ val killed = intercept[SparkException] {
+ ParquetStorageFilter.rethrowIfMustPropagate(under(1, oom))
+ }
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ } finally {
+ TaskContext.unset()
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ TaskContext.setTaskContext(TaskContext.empty())
+ val thrown = try {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ } finally {
+ TaskContext.unset()
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty)
+ .getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ val (_, sevenUncapped, _) = run("64MB", 394L)
+ val (_, sevenAtCharge, _) = run("103b", 394L)
+ val (_, sevenUnderCharge, _) = run("102b", 394L)
+ assert(sevenAtCharge == sevenUncapped,
+ s"a cap of exactly the charge must keep splicing; got $sevenAtCharge against " +
+ sevenUncapped)
+ assert(sevenUnderCharge > sevenUncapped,
+ s"a cap one byte under it must give splicing up; got $sevenUnderCharge against " +
+ sevenUncapped)
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a later accumulator set is sized up front only when its charge fits under the cap") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, which takes that memory before any value arrives, so it is charged then. Six survivors
+ // of a 1000-byte key over a capacity of 4 fill one set of four, charged about 16KB, and sizing
+ // the next set would charge as much again for two survivors. Under a cap of 28000 bytes that
+ // does not fit, so the set grows value by value instead, and the row group stays spliced, which
+ // reading the key column only once shows. The sizing has to fit after the fifth survivor's own
+ // fixed 17 bytes too. A cap of 32120 leaves 15995 for it, so the set grows and stays spliced.
+ // Under a cap that fits it, the second set is sized and charged once. A cap of 32130 holds
+ // the 32125 bytes of the fifth survivor and gives splicing up at the sixth, at 32142. A cap
+ // of 40000 keeps splicing. A missing or a double charge would each get one of these wrong.
+ // And the keys that land in the second set must come back right either way.
+ withTempDir { dir =>
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), REPEAT('x', 996)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val fileSchema = kvSchema(keyType = StringType)
+ val storageFilters = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, fileSchema, storageFilters, maxSplicedBytes, batchSize = 4)
+
+ val (uncappedRows, uncappedBytes) = run("64MB")
+ val (cappedRows, cappedBytes) = run("28000b")
+ assert(uncappedRows == 6 && cappedRows == 6,
+ s"the filter keeps keys 15..20 either way; got $uncappedRows and $cappedRows")
+ assert(cappedBytes == uncappedBytes,
+ "a set whose sizing does not fit must grow instead, so the row group stays spliced and " +
+ s"reads the key column once; capped=$cappedBytes uncapped=$uncappedBytes")
+ val (_, noRoomBytes) = run("32120b")
+ assert(noRoomBytes == uncappedBytes,
+ "a sizing that fits only without this survivor's own charge must not happen, so a cap of " +
+ s"32120 bytes keeps splicing; got $noRoomBytes against $uncappedBytes")
+ val (_, justUnderBytes) = run("32130b")
+ assert(justUnderBytes > uncappedBytes,
+ "the sizing must be charged when the set is sized, which passes a cap of 32130 bytes; " +
+ s"got $justUnderBytes against $uncappedBytes")
+ val (_, roomyBytes) = run("40000b")
+ assert(roomyBytes == uncappedBytes,
+ "and charged once, so a cap of 40000 bytes keeps splicing; " +
+ s"got $roomyBytes against $uncappedBytes")
+
+ val expected = (15 to 20).map(id => (f"$id%04d" + "x" * 996, id.toString))
+ Seq(Long.MaxValue, 28000L).foreach { cap =>
+ val (pairs, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(storageFilters, fileSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString),
+ capacity = 4)
+ try {
+ assert(pairs == expected,
+ s"every survivor must come back with its own key under cap $cap; got " +
+ pairs.map(_._2))
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ val outDir = new File(dir, "damaged").getAbsolutePath
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ rows.toDF("k", "v").repartition(1).write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 64L * 1024 * 1024)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .option(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT, 100L)
+ .parquet(outDir)
+ val path = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ .head.getAbsolutePath
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(false, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(false, 16, storageFilter)
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ (read.toSeq, error)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError) = readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ }
+ }
+
+ test("a key value the filter does not read and cannot be decoded gives splicing up") {
+ // The filter need not read every key on every row, as `coalesce(a, b)` reads `b` only where
+ // `a` is null. The copy into the survivor buffer reads every key, so it can meet a value no
+ // plain read decodes. A dictionary-encoded column decodes a value only when it is read, so a
+ // dictionary that holds fewer values than its ids point at is such a value. Copying it gives
+ // splicing up for the row group, and phase 2 reads the key columns the way a plain read does.
+ // So the read answers where a plain one does, and the filter still applies.
+ withTempDir { dir =>
+ // Two copies of each `b`, so parquet keeps its dictionary. Cut to 64 values, it leaves `b`
+ // undecodable from row 128 on. `a` is null only before that, so the filter never reads a
+ // damaged `b`.
+ val rows = (0 until 200).map { i =>
+ (if (i < 100 && i % 5 == 0) null else s"a_$i", s"b_${i / 2}", i)
+ }
+ val outDir = new File(dir, "dictionary").getAbsolutePath
+ rows.toDF("a", "b", "v").repartition(1).write
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "true")
+ .parquet(outDir)
+ val path = new File(outDir).listFiles((_, name) => name.endsWith(".parquet")).head
+ .getAbsolutePath
+ assert(encodingsOf(path, "b").exists(_.usesDictionary),
+ s"the fixture's whole point; got ${encodingsOf(path, "b")}")
+ shrinkDictionary(path, column = "b", entries = 64)
+ val columns = Seq("a", "b", "v")
+ intercept[ArrayIndexOutOfBoundsException] {
+ readAllWith(path, columns, null, (b, i) => b.column(1).getUTF8String(i).toString)._2
+ .close()
+ }
+
+ val schema = StructType(Seq(
+ StructField("a", StringType), StructField("b", StringType), StructField("v", IntegerType)))
+ val coalesced = Coalesce(Seq(BoundReference(0, StringType, nullable = true),
+ BoundReference(1, StringType, nullable = true)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(coalesced, Literal.create("a_15", StringType))), schema)
+ val ((result, reader), givenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, columns, filter, (b, i) => b.column(2).getInt(i), capacity = 16)
+ }
+ try {
+ // Keeps rows 150-199 among others, whose `b` is damaged.
+ val expected = rows.filter { case (a, b, _) => Option(a).getOrElse(b) >= "a_15" }.map(_._3)
+ assert(expected.contains(150), "the filter must keep a row whose `b` is damaged")
+ assert(result == expected, s"got ${result.size} rows, expected ${expected.size}")
+ assert(givenUp, "the copy must have failed and given splicing up, or this proves nothing")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry reads the damaged row group whole over the pushed data filter's ranges, and only
+ // that one, since parquet keeps a column index store per block. What it must not give up is
+ // the pushed data filter's own page pruning, which a plain read of the same query keeps, nor
+ // the other row groups' narrowing. Dropping either would read more than it needs to.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warned) = withPartialReadWarning(read(evenKeys))
+ assert(warned, "the read must have discovered the damaged index, or this test proves nothing")
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ // The row groups around the damaged one keep the storage filter's narrowing too, so they
+ // return even keys alone.
+ val undamaged = filtered.filter(k => k <= 100L || k > 200L)
+ assert(undamaged.nonEmpty && undamaged.forall(_ % 2 == 0),
+ s"the other row groups must still be narrowed; got ${undamaged.filter(_ % 2 != 0)}")
+ }
+ }
+
+ test("a v2 file's DELTA_BYTE_ARRAY decimal column is read over scattered survivors") {
+ // Phase 2 skips rows inside a page whenever survivors scatter, where a plain read skips only
+ // when a pushed filter cuts a page. The parquet v2 writer stores a FIXED_LEN_BYTE_ARRAY
+ // decimal without a dictionary as DELTA_BYTE_ARRAY, whose reader could not skip until
+ // SPARK-59828, so this read failed where a plain read of the same file did not.
+ withTempDir { dir =>
+ val outDir = new File(dir, "v2").getAbsolutePath
+ spark.range(0, 1000).selectExpr("id AS k", "CAST(id * 1.25 AS DECIMAL(30, 2)) AS d")
+ .repartition(1)
+ .write
+ .option("parquet.writer.version", "PARQUET_2_0")
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .parquet(outDir)
+ val path = new File(outDir).listFiles((_, name) => name.endsWith(".parquet")).head
+ .getAbsolutePath
+ assert(encodingsOf(path, "d").contains(Encoding.DELTA_BYTE_ARRAY),
+ s"the fixture's whole point; got ${encodingsOf(path, "d")}")
+ val everyThirdRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(3L)), Literal(0L))
+ val schema = kvSchema(valueType = DecimalType(30, 2))
+ val (result, reader) = readAllWith(path, Seq("k", "d"),
+ createFilter(Seq(everyThirdRow), schema),
+ (b, i) => (b.column(0).getLong(i), b.column(1).getDecimal(i, 30, 2).toJavaBigDecimal))
+ try {
+ val expected = (0L until 1000L by 3L)
+ .map(k => (k, new java.math.BigDecimal(k).multiply(new java.math.BigDecimal("1.25"))
+ .setScale(2)))
+ assert(result == expected, s"got ${result.take(5)} (${result.size} rows)")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the survivor queue and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ val fileSchema = kvSchema()
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAll(path, filter, capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // Each survivor costs 9 bytes of long key and 40 of its own range. The sum passes the cap of
+ // 1024 at survivor 21, which releases the buffer, and the ranges alone at survivor 26, which
+ // gives the filter up. The capacity decides what the buffer holds when it is released. At
+ // 4096 that is one partly filled set of accumulators, and at 16 a full set in the queue as
+ // well. Both have to end with the queue empty and the filter given up. Keeping one without
+ // the other would leave phase 2 reading non-key columns while emit spliced keys from a queue
+ // holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAll(path, filter)
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSeq == Seq("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does, since the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers conjuncts under the relation's own column names") {
+ // The analyzer spells a reference the way the query did, so under case-insensitive resolution
+ // a bloom can name `_TMP_METADATA_ROW_INDEX` while the relation, and the schema the reader
+ // requests, call the column `_tmp_metadata_row_index`. The planner normalizes what it offers to
+ // the relation's names, so the format's gate sees the name the reader will see and declines.
+ // Without that the gate passed it, `ParquetStorageFilter.create` rejected it on the driver, and
+ // a query that runs with the conf off failed with it on.
+ withTempDir { dir =>
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", s"id AS $rowIndexName", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", rowIndexName, "v")
+ .queryExecution.optimizedPlan
+ def column(name: String): Attribute =
+ relation.output.find(_.name == name).getOrElse(fail(s"no $name in the relation output"))
+
+ def offered(key: Attribute): Seq[Expression] = {
+ val logical = LogicalFilter(
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ scanOf(physical).storageFilters
+ }
+
+ // As a control, an ordinary key spelled in another case is offered, under its own name.
+ val upperK = offered(column("k").withName("K"))
+ assert(upperK.size == 1, s"a bloom over `K` must reach the scan; got $upperK")
+ assert(upperK.head.references.map(_.name).toSet == Set("k"),
+ s"and it must carry the relation's name for the column; got ${upperK.head}")
+
+ val upperRowIndex =
+ offered(column(rowIndexName).withName(rowIndexName.toUpperCase(Locale.ROOT)))
+ assert(upperRowIndex.isEmpty,
+ s"a bloom over the row-index column must not be offered in any case; got $upperRowIndex")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for. A dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`, since `/` is floating-point division in Spark SQL, which would hand every
+ // key expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize, dictionary = dictionary)
+ }
+
+ // Every column chunk of the file, from its footer. The facts the tests below assert about a
+ // fixture are read off these rather than assumed from the write options.
+ private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] =
+ footerOf(filePath).getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq
+
+ private def footerOf(filePath: String): ParquetMetadata = {
+ val conf = spark.sessionState.newHadoopConf()
+ ParquetFooterReader.readFooter(
+ HadoopInputFile.fromPath(new Path(filePath), conf), ParquetMetadataConverter.NO_FILTER)
+ }
+
+ // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String, column: String = "k"): Set[Encoding] =
+ footerChunks(filePath).filter(_.getPath.toDotString == column)
+ .flatMap(_.getEncodings.asScala).toSet
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // builds the plain vectorized reader rather than the late-materialization one. Every read helper
+ // in this suite goes through here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need. The caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ if (storageFilter == null) {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } else {
+ new LateMaterializationParquetRecordReader(useOffHeap, capacity, storageFilter)
+ }
+ } { reader =>
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // A key value as a string in its *internal* representation, so the splicing path and the plain
+ // path can be compared without knowing the type and without external type conversion.
+ // `InternalRow.getAccessor` does the per-type read, which is why this takes the batch's row
+ // rather than the column vector.
+ private def renderValue(row: InternalRow, ordinal: Int, dt: DataType): String =
+ InternalRow.getAccessor(dt)(row, ordinal) match {
+ case null => "null"
+ // Two equal arrays are two objects, so the default toString would compare identities.
+ case bytes: Array[Byte] => bytes.mkString(",")
+ case value => value.toString
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path. Whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.getRow(i), 0, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage, one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why. There is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ } {
+ val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded"
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ // A copier that fails gives splicing up, and phase 2 then reads the keys plainly, so the
+ // rows alone would not show it.
+ val ((result, reader), splicingGivenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ }
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ assert(!splicingGivenUp, s"every $name survivor must be copied into the buffer")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // This is a regression test. TimeType is an AtomicType that passes every planning-time gate
+ // (it is batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a
+ // TIME join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // What FileSourceStrategy offers, what ParquetStorageFilter.create admits and what the reader
+ // can copy are one list, because isSupportedKeyType answers from ValueCopier. Both are asserted
+ // here. A type the copier has no case for would be a task failure rather than a planning-time
+ // rejection, and the delegation is what rules that out.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ assert(ValueCopier.forType(dt) != null, s"$dt should have a value copier")
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ val e = intercept[SparkException](ValueCopier.forType(dt))
+ assert(e.isInternalError && !DataSourceUtils.shouldIgnoreCorruptFileException(e),
+ s"an unsupported key type is the reader's bug, which must fail loudly; got $e")
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, which fixes the key row's
+ // layout. The reader pairs each key column's batch slot with its key-row position, so nothing
+ // depends on that order. Every other fixture mentions its keys in column order, and production
+ // does not promise that, since the conjuncts arrive in `afterScanFilters` order, which follows
+ // the join. So this has two key columns mentioned the other way round, and asserts both the
+ // sorted layout and that every key still comes back with its own row.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ keyAtLeast(100L))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // The emit path rewrites the key slots alone, and a partition slot is not one of them. It sits
+ // past the data columns and must keep the constant vector `initBatch` populated for it.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite. OffHeapColumnVector.close() frees
+ // the native buffer, so a read after close is a crash rather than stale data. A second close
+ // is harmless, since close() zeroes the addresses it freed. capacity = 16 over 71 survivors
+ // forces 5 emits, each closing the previous emit's dequeued key vectors. The other 29 rows
+ // are what makes a declining reader fail this.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = keyAtLeast(30L)
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (30L to 100L),
+ s"expected the 71 surviving keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // One ColumnarBatch is handed out for the whole read and its key slots are rewritten per
+ // batch, so a consumer holding on to an earlier getCurrentValue() would read the wrong
+ // vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new LateMaterializationParquetRecordReader(false, 8, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution, a column missing from the physical file -----
+
+ test("non-key column missing from a file: the evolved table reads and credits avoided bytes") {
+ // Schema evolution end to end, with the non-key column rather than the key missing from the
+ // older file. The clipped requested schema keeps the missing column, which the byte walks skip
+ // by themselves, since they go over the block's chunks and the older file has no chunk for it.
+ // What this pins is that such a read returns the right rows and still credits a saving.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // The older file has (k, v), the newer one (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSeq.sorted
+ val expected = (150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // A positive total says the byte walks ran over the older file's row groups as well.
+ // Asserting non-negativity would prove nothing. `SQLMetric.add` drops a negative and the
+ // value cannot go below zero.
+ val bytesRg = withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value + bytesPf.value > 0,
+ s"the walk must credit some avoided bytes; got rg=${bytesRg.value} pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = scanOf(spark.read.parquet(path).select("k", "v").queryExecution.executedPlan)
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Seq[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read. The rows come
+ // back sorted, so a row returned twice shows.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Seq[(Long, Option[Long])]) = {
+ val withSF = withStorageFilters(spark.sql(s"SELECT id, k FROM $table")) { attr =>
+ Seq(buildFilter(attr("k")))
+ }
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSeq.sorted
+ (withSF, collected)
+ }
+
+ test("all keys missing: an error on the constant reads the file rather than skipping it") {
+ // A file with none of the filter's key columns makes the predicate constant over it, so the
+ // reader decides once, from the values `initBatch` materialized for those columns. That
+ // evaluation can throw exactly as a row's value can. Here the older file's `k` reads as its
+ // DEFAULT, which is not a number, and the cast fails under ANSI. Failing open reads the file
+ // the way a plain scan would, so its rows survive. Skipping it would lose them, and rethrowing
+ // would fail a query a plain read answers.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true",
+ SQLConf.ANSI_ENABLED.key -> "true") {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql("ALTER TABLE evolved ADD COLUMN k STRING DEFAULT 'not-a-number'")
+ spark.sql("INSERT INTO evolved VALUES (4, '40'), (5, '50')")
+ val withSF = withStorageFilters(spark.sql("SELECT id, k FROM evolved")) { attr =>
+ Seq(GreaterThanOrEqual(Cast(attr("k"), LongType), Literal(1L)))
+ }
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val ids = collectRows(withSF).map(_.getLong(idPos)).toSeq.sorted
+ assert(ids == Seq(1L, 2L, 3L, 4L, 5L),
+ s"the older file must fail open and the newer one pass the cast; got $ids")
+ }
+ }
+ }
+
+ test("all keys missing: a checked exception on the constant reads the file") {
+ // The constant evaluation fails open under the same rule as a row's, a checked exception
+ // included. The older file's `k` reads as its DEFAULT 7, which the key expression throws on,
+ // so that file must be read the way a plain scan would. The newer file's keys evaluate
+ // normally and are rejected, since no Filter runs above this scan.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(OnKey(k, at = 7L, error = new SparkException(_)),
+ Literal(100L)))
+ assert(collected == Seq((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L))),
+ s"the older file must fail open and the newer one be filtered; got $collected")
+ }
+ }
+ }
+
+ test("a file with none of the projected non-key columns is read plainly") {
+ // The clipped requested schema keeps a column the file does not have, so a projected non-key
+ // column added after the older file was written still looks like something for phase 2 to
+ // read. Phase 2 reads nothing for it, so for that file the filter could only add cost, and the
+ // reader declines it there. With no Filter above this scan, the older file's rows come back
+ // unfiltered while the newer file's are still filtered.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTable("evolved_value") {
+ spark.sql("CREATE TABLE evolved_value (k BIGINT, v STRING) USING parquet")
+ spark.sql("INSERT INTO evolved_value VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+ spark.sql("ALTER TABLE evolved_value ADD COLUMN w STRING")
+ spark.sql("INSERT INTO evolved_value VALUES (4, 'd', 'x'), (5, 'e', 'y')")
+ val withSF =
+ scanWithStorageFilter(spark.sql("SELECT k, w FROM evolved_value"), threshold = 5L)
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val keys = collectRows(withSF).map(_.getLong(kPos)).toSeq.sorted
+ assert(keys == Seq(1L, 2L, 3L, 5L),
+ s"the older file must be read unfiltered and the newer one filtered; got $keys")
+ }
+ }
+ }
+
+ test("a file with none of a projected struct's requested fields is read plainly") {
+ // Under the legacy returnNullStructIfAllFieldsMissing, the clipped schema keeps a struct none
+ // of whose requested fields the file has, and adds no field it has, so phase 2 would read
+ // nothing for it. Only the struct's leaves are missing columns, not the struct, so the reader
+ // asks the file schema about every non-key leaf. The filter rejects every row, so a declined
+ // file comes back whole and one that is not comes back empty.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 20).selectExpr("id AS k", "named_struct('a', CAST(id AS INT)) AS s"),
+ rowGroupSize = 64 * 1024L)
+ val schema =
+ new StructType().add("k", LongType).add("s", new StructType().add("b", IntegerType))
+ val storageFilters = Seq(keyAtLeast(1000L))
+ withSQLConf(SQLConf.LEGACY_PARQUET_RETURN_NULL_STRUCT_IF_ALL_FIELDS_MISSING.key -> "true") {
+ val keys = readLongs(formatReader(schema, Nil, storageFilters), path, ordinal = 0)
+ assert(keys == (0L until 20L), s"the file must be read plainly; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Seq[(Long, Option[Long])] =
+ Seq((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test. The default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Seq[(Long, Option[Long])] = Seq((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected, s"got $collected; expected $expected")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(StorageFilterMetrics.ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Seq((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got $rejectsNull")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Seq[(Long, Option[Long])] =
+ Seq((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got $acceptsNull")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ val collected = collectRows(withSF).map(r => (r.getLong(0), r.getLong(2))).toSeq.sorted
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1))
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got ${collected.take(5)}")
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF =
+ scanWithStorageFilter(spark.read.parquet(outDir).select("k", "s"), threshold = 195L)
+ val collected = collectRows(withSF)
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSeq.sorted
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k"))
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ test("a storage filter binds to the right column with partition and metadata columns in output") {
+ // `preparedStorageFilters` binds the filter's references against `output.take(requiredSchema
+ // .length)`, and `output` is the data columns, then the generated metadata columns, then the
+ // partition columns, then the constant metadata ones. A projection carrying all four is where a
+ // wrong prefix would bind the predicate to another column and drop rows the filter keeps, which
+ // the post-scan Filter cannot put back. The file's first column `a` is not projected, so the
+ // key sits at a different position in the relation's data schema than in the scan's.
+ withTempDir { dir =>
+ val outDir = new File(dir, "parted").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id * 7 AS a", "id AS k", "CONCAT('v_', id) AS v", "CAST(id % 3 AS INT) AS p")
+ .repartition(1)
+ .write
+ .partitionBy("p")
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select(
+ col("k"), col("v"), col("p"),
+ col("_metadata.row_index").as("ri"), col("_metadata.file_size").as("fs"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ // The assumption the binding rests on, stated where a change to `output` would break it.
+ assert(withSF.output.take(withSF.requiredSchema.length).map(_.name) ==
+ withSF.requiredSchema.fieldNames.toSeq,
+ s"the scan's leading attributes must be its data columns; got " +
+ s"${withSF.output.map(_.name)}")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val pPos = withSF.output.indexWhere(_.name == "p")
+ def collectAll(plan: SparkPlan): Seq[Seq[Any]] =
+ collectRows(plan).map(_.toSeq(plan.output.map(_.dataType))).toSeq
+ .sortBy(_(kPos).asInstanceOf[Long])
+ val filtered = collectAll(withSF)
+ // The same scan without the filter is the oracle, so the row indexes and file sizes come
+ // from the read rather than from arithmetic over how the partitions were written.
+ val expected = collectAll(withSF.copy(storageFilters = Nil))
+ .filter(_(kPos).asInstanceOf[Long] >= 150L)
+ assert(filtered == expected,
+ s"expected ${expected.size} rows with their own partition and metadata values; " +
+ s"got ${filtered.size}")
+ assert(filtered.map(_(pPos)).distinct.size == 3,
+ s"every partition value must still be represented; got ${filtered.map(_(pPos)).distinct}")
+ }
+ }
+ }
+
+ // ----- Planner gates -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false. It may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ //
+ // This hook answers about shapes and types alone. Whether the conjunct is deterministic and
+ // whether its references are projected columns of this scan are plan-visible, so they stay with
+ // the planner; `FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter`
+ // covers that half.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ // A cast key is supported. In ANSI mode it can throw on a row an earlier conjunct would have
+ // dropped, and the reader handles that where it arises, by giving the filter up for the row
+ // group. Declining here instead would also decline every widening cast, which is what type
+ // coercion inserts for a join between an int and a bigint column.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+
+ // The conf is read in the format rather than in the planner, and it is the per-scan question
+ // that carries it, asked once before anything per conjunct.
+ assert(!format.supportsStorageFilterPushdown(spark), "the feature is off with the conf off")
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ assert(format.supportsStorageFilterPushdown(spark), "and on with it on")
+ assert(!subclass.supportsStorageFilterPushdown(spark), "a subclass must not claim support")
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ assert(format.supportsStorageFilter(onCast),
+ "a cast key is pushed, and an evaluation error gives the row group up")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The rows are right either way, since the bloom stays in the post-scan Filter. What this gate
+ // saves is offering the scan a filter its reader cannot apply.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one. Every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSeq.sorted
+ assert(keys == (1L to 50L),
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support declines rather than delegating") {
+ // The default `FileFormat.buildReaderWithStorageFilters` answers None, and the caller falls
+ // back to the ordinary builder. That is what keeps the two builders from being able to call
+ // each other, and declining is safe because the post-scan Filter still holds the conjunct.
+ val storageFilters = Seq(keyAtLeast(1L))
+ val declined = new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration(), Map.empty)
+ assert(declined.isEmpty, "the default must not build a reader of its own")
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilterPushdown(spark),
+ "and it must not claim support either")
+ }
+
+ test("a scan whose format declines falls back to the ordinary reader and returns every row") {
+ // The check above is that the default answers None. This is the other half, that `inputRDD`
+ // then builds the ordinary reader and the scan still produces its rows. A `ParquetFileFormat`
+ // subclass is the shape that reaches it, since it inherits the override and declines by class,
+ // and its ordinary builder is the one an extension would have customized.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, threshold = 195L)
+ val declining = withSF.copy(
+ relation = withSF.relation.copy(fileFormat = new ParquetFileFormat() {})(spark))
+ val collected = executePlanCollect(declining).toSeq.sorted
+ assert(collected == rows,
+ s"a declining format reads the file plainly; got ${collected.size} of ${rows.size} rows")
+ }
+ }
+ }
+
+ Seq(false, true).foreach { aqe =>
+ test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") {
+ // AQE is on by default in production, and it is where the bloom subquery is planned by
+ // PlanAdaptiveSubqueries rather than PlanSubqueries, which is the path
+ // preparedStorageFilters' ScalarSubquery materialization depends on.
+ withBloomFilterTables {
+ val baseConf = Map(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString)
+ def run(pushdown: Boolean): (Int, Seq[(Long, Long)], Long) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, rows) = runBloomFilterJoin()
+ val excluded = collect(plan) {
Review Comment:
Done. One `rowsExcludedByStorageFilters` helper beside `countBloomFiltersInStorageFilters` serves all three, through `collect(plan)`. The codegen-off test takes its scans through `collect(plan)` too, and requires at least one.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether a log line containing `text` was logged.
+ private def withLogged[T](text: String)(body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains(text))
+ (result.get, logged)
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) =
+ withLogged("without page-level storage filtering")(body)
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def readWith(key: Expression): SparkException = intercept[SparkException] {
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ readAll(path, filter)._2.close()
+ }
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException = {
+ TaskContext.setTaskContext(TaskContext.empty())
+ try readWith(key) finally TaskContext.unset()
+ }
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_))),
+ classOf[SparkException])
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ val key = OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped")))
+ readAll(path, createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested))._2
+ .close()
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in a checked wrapper, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ assertPropagates(readWith(OnKey(bound, at = 7L, error = new InternalError(_))),
+ classOf[InternalError])
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val assertingKey = OnKey(bound, at = 7L, error = new AssertionError(_))
+ val (result, reader) =
+ readAll(path, createFilter(Seq(GreaterThanOrEqual(assertingKey, Literal(10L))), requested))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagate(under(depth - 1, oom))
+ } eq oom, "the deepest level the executor sees counts")
+ ParquetStorageFilter.rethrowIfMustPropagate(under(depth, oom))
+
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(depth - 2, oom))
+ } eq oom, "a decoding error's chain counts one level less deep")
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(depth - 1, oom))
+
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(1, new InternalError("codec")))
+ // One from an expression goes out in a checked wrapper, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file, and the executor finds it under it.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[SparkException](ParquetStorageFilter.rethrowIfMustPropagate(e))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ assert(Executor.isFatalError(thrown, depth), s"got $thrown")
+ }
+ ParquetStorageFilter.rethrowIfMustPropagate(new AssertionError("not fatal"))
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM.
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(growth)
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagate(growth)
+ }.getMessage == "reserve")
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.setTaskContext(context)
Review Comment:
Done, all three use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4187892150
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -476,34 +483,99 @@ private boolean containsPath(Type parquetType, String[] path, int depth) {
return false;
}
- private void checkEndOfRowGroup() throws IOException {
- if (rowsReturned != totalCountLoadedSoFar) return;
+ /**
+ * Loads the row group the read continues with, and returns false if there is none left. Only a
+ * subclass that picks its own rows can answer false, because this class reads every row group
+ * parquet hands it, and then {@code totalRowCount} already says when they run out.
+ */
+ protected boolean loadNextRowGroup() throws IOException {
PageReadStore pages = reader.readNextRowGroup();
if (pages == null) {
throw new IOException("expecting more rows but reached last block. Read "
+ rowsReturned + " out of " + totalRowCount);
}
+ // Null ranges, because this path never picks the rows itself, and whether parquet narrowed the
+ // row group by a pushed filter's column index is known only to the store it handed back.
+ installRowGroup(pages, null);
+ return true;
+ }
+
+ /**
+ * Points the column readers at one row group's {@code pages}, from which they are to read the
+ * rows {@code rowRanges} names. Null ranges leave that to the store, which names its rows through
+ * its row indexes when parquet filtered it and otherwise holds every row of the row group.
+ */
+ protected final void installRowGroup(PageReadStore pages, RowRanges rowRanges)
+ throws IOException {
if (rowIndexGenerator != null) {
rowIndexGenerator.initFromPageReadStore(pages);
}
- for (ParquetColumnVector cv : columnVectors) {
- initColumnReader(pages, cv);
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (suppliesOwnVector(i)) {
+ // Cleared rather than left alone, because a reader a previous row group set would otherwise
+ // be driven over this row group's pages.
+ columnVectors[i].setColumnReader(null);
+ } else {
+ initColumnReader(pages, rowRanges, columnVectors[i]);
+ }
}
totalCountLoadedSoFar += pages.getRowCount();
}
- private void initColumnReader(PageReadStore pages, ParquetColumnVector cv) throws IOException {
+ /**
+ * Drops the column readers of the row group installed last, and the row-index iterator, both of
+ * which hold the row ranges that row group was read over. For a subclass that does work of its
+ * own between two row groups, so that work does not run with the previous ranges still held.
+ * Call it only from {@link #loadNextRowGroup}, once the row group is fully emitted. A batch read
+ * after it needs another row group installed first.
+ */
+ protected final void releaseRowGroupReaders() {
Review Comment:
Thanks, both updated. The description says 615 lines, seven protected methods with five of them new, and that only the subclass calls `releaseRowGroupReaders`. The narrow-table line now says the scan takes up to 1.23x the time.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
Review Comment:
Done, both, and the `ResolveDefaultColumns` import is gone.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
+ try {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ rethrow(under(2, oom), fromRead = false)
+ } finally {
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, depth.toString)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ rethrow(new InternalError("codec"), fromRead = true)
+ rethrow(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ rethrow(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ rethrow(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val withDefault = new MetadataBuilder()
+ .putString(ResolveDefaultColumns.EXISTS_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .putString(ResolveDefaultColumns.CURRENT_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .build()
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = true, withDefault), StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split") {
+ // Two row groups give the filter up on an evaluation error, and the reader reports that once.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val rowGroups = footerOf(path).getBlocks.asScala.map(_.getRowCount)
+ assert(rowGroups.size == 2 && rowGroups.head >= 5L && rowGroups.head < 150L,
+ s"keys 5 and 150 must fall in two row groups; got $rowGroups")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val filter = createFilter(Seq(And(failingAt(5L), failingAt(150L))), kvSchema())
+ val ((result, reader), warnings) =
+ withLogged("applying it to a row raised an error")(readAll(path, filter))
+ try {
+ assert(result == rows, s"both row groups must come back whole; got ${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ val (_, sevenUncapped, _) = run("64MB", 394L)
+ val (_, sevenAtCharge, _) = run("103b", 394L)
+ val (_, sevenUnderCharge, _) = run("102b", 394L)
+ assert(sevenAtCharge == sevenUncapped,
+ s"a cap of exactly the charge must keep splicing; got $sevenAtCharge against " +
+ sevenUncapped)
+ assert(sevenUnderCharge > sevenUncapped,
+ s"a cap one byte under it must give splicing up; got $sevenUnderCharge against " +
+ sevenUncapped)
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the rows left and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the rows that can still reach it. That memory is taken before any value
+ // arrives, so it is charged then. Six survivors over a capacity of 4 fill one set of four
+ // 1000-byte keys, and the second set starts two rows before the end of the row group. So it
+ // is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000 bytes
+ // each, so they add 8000 more once the sizing is used up, and the row group holds 32142 bytes
+ // in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or a
+ // double charge would each get wrong. The second set's byte child is reserved for 2000 bytes,
+ // which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000. And
+ // the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)
+
+ val (uncappedRows, uncappedBytes) = run("64MB")
Review Comment:
Done. One `assertSpliceCharge(charge, rows)(read)` serves the three. `withPartialReadWarning` is gone, so those checks assert the count. And `readWith` is inlined.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
+ try {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ rethrow(under(2, oom), fromRead = false)
+ } finally {
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, depth.toString)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ rethrow(new InternalError("codec"), fromRead = true)
+ rethrow(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ rethrow(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ rethrow(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val withDefault = new MetadataBuilder()
+ .putString(ResolveDefaultColumns.EXISTS_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .putString(ResolveDefaultColumns.CURRENT_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .build()
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = true, withDefault), StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split") {
+ // Two row groups give the filter up on an evaluation error, and the reader reports that once.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val rowGroups = footerOf(path).getBlocks.asScala.map(_.getRowCount)
+ assert(rowGroups.size == 2 && rowGroups.head >= 5L && rowGroups.head < 150L,
+ s"keys 5 and 150 must fall in two row groups; got $rowGroups")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val filter = createFilter(Seq(And(failingAt(5L), failingAt(150L))), kvSchema())
+ val ((result, reader), warnings) =
+ withLogged("applying it to a row raised an error")(readAll(path, filter))
+ try {
+ assert(result == rows, s"both row groups must come back whole; got ${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ val (_, sevenUncapped, _) = run("64MB", 394L)
+ val (_, sevenAtCharge, _) = run("103b", 394L)
+ val (_, sevenUnderCharge, _) = run("102b", 394L)
+ assert(sevenAtCharge == sevenUncapped,
+ s"a cap of exactly the charge must keep splicing; got $sevenAtCharge against " +
+ sevenUncapped)
+ assert(sevenUnderCharge > sevenUncapped,
+ s"a cap one byte under it must give splicing up; got $sevenUnderCharge against " +
+ sevenUncapped)
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the rows left and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the rows that can still reach it. That memory is taken before any value
+ // arrives, so it is charged then. Six survivors over a capacity of 4 fill one set of four
+ // 1000-byte keys, and the second set starts two rows before the end of the row group. So it
+ // is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000 bytes
+ // each, so they add 8000 more once the sizing is used up, and the row group holds 32142 bytes
+ // in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or a
+ // double charge would each get wrong. The second set's byte child is reserved for 2000 bytes,
+ // which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000. And
+ // the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)
+
+ val (uncappedRows, uncappedBytes) = run("64MB")
+ val (atChargeRows, atChargeBytes) = run("32142b")
+ assert(uncappedRows == 6 && atChargeRows == 6,
+ s"the filter keeps keys 15..20 either way; got $uncappedRows and $atChargeRows")
+ assert(atChargeBytes == uncappedBytes,
+ "a cap of exactly the row group's charge must keep splicing and read the key column " +
+ s"once; got $atChargeBytes against $uncappedBytes")
+ val (_, underChargeBytes) = run("32141b")
+ assert(underChargeBytes > uncappedBytes,
+ "a cap one byte under it must give splicing up and read the key column again; " +
+ s"got $underChargeBytes against $uncappedBytes")
+
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ if (cap >= 32142L) {
+ assert(secondSet == Seq(4000),
+ s"the second set must be sized for the two values left under cap $cap; got $secondSet")
+ }
+ }
+ }
+ }
+
+ test("a later accumulator set is sized only when it fits with its rows' fixed part") {
+ // The second set's two keys are 500 bytes here, shorter than the first set's 1000, so growing
+ // value by value is the cheaper path: 20142 bytes in all against 24142 sized. A cap of 24142
+ // sizes the set, whose byte child is then reserved for 2000 bytes and holds 4000. One of
+ // 24141 leaves 8033 bytes after the first set and its range, a byte short of the 8000 of
+ // sizing and the 34 of the two rows' fixed part. So the set grows instead, to a 1000-byte
+ // child, and keeps splicing. A cap of 20141 gives splicing up.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 500)
+ def run(maxSplicedBytes: String): Long =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)._2
+
+ val uncapped = run("64MB")
+ assert(run("24142b") == uncapped, "a cap that fits the sizing must keep splicing")
+ assert(run("24141b") == uncapped, "and so must one that only fits growing value by value")
+ assert(run("20141b") > uncapped, "a cap under both must give splicing up")
+
+ Seq(Long.MaxValue -> 4000, 24142L -> 4000, 24141L -> 1000).foreach { case (cap, child) =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected, s"every survivor must come back with its own key under cap $cap")
+ assert(secondSet == Seq(child),
+ s"under cap $cap the second set's byte child must hold $child bytes; got $secondSet")
+ }
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, schema, storageFilters, maxSplicedBytes, batchSize = 16)
+ val (_, uncapped) = run("64MB")
+ val (rows, atCharge) = run("166b")
+ val (_, underCharge) = run("165b")
+ assert(rows == 7, s"the filter keeps keys 394..400; got $rows")
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Two row groups of 100 rows with `v = k * 10`, and a pushed `v IN (x, 1500)`, where `x` falls
+ // between the first two pages of the first row group. Its statistics keep that row group, while
+ // its column index keeps no page of it.
+ private def writeColumnIndexEmptiedFile(dir: File): (String, Seq[sources.Filter]) = {
Review Comment:
Done, both moved.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTestAccess.scala:
##########
@@ -173,17 +187,74 @@ object ParquetTestAccess {
try { deltaLengthByteArrayCtor.newInstance() }
catch { case e: ReflectiveOperationException => throw rethrow(e) }
+ // -------- LateMaterializationParquetRecordReader --------
+
+ private val keyScratchVectorsField = {
Review Comment:
Done, they are lazy now, with a comment saying why.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-6006981501
Thanks @peter-toth ! The design looks solid to me. In fact we also did the same in our internal DataFusion Comet and plan to upstream it later too. Full review coming soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-6064730845
Thanks for landing this, @peter-toth. A few follow-ups after reading the merged version. None of them is a correctness concern.
**Where the feature can pay off in practice**
I think the set of real workloads this helps is narrower than the benchmark suggests, and it would help to say so, or to measure it, before anyone considers changing the default.
- **The bloom has to exist first.** `InjectRuntimeFilter` only injects one when the join is probably a shuffle join, or the application side already has a shuffle below it, and the application-side scan is at least 10GB by default. A typical fact table broadcast-joined with a filtered dimension gets no bloom, so it gets nothing from this PR.
- **The survivors need physical locality.** A page is the unit that can be skipped, and with default writer settings a narrow column's page holds up to 20,000 rows. With survivors spread at random at rate p, a page has no survivor with probability about e^(-p·20000), so page skipping needs p well below 1/20,000, and the bloom's false positives count toward p. In practice the gain comes from row groups or runs of pages with no survivor at all. That needs the surviving keys to form a range in the table's physical order, or each key to own whole pages. Hash bucketing does not give that. The most natural candidate, a date key, is usually handled already: by DPP when the table is partitioned on it, and by broadcast when the date dimension is small.
- **The tables most likely to be clustered on a join key are out of reach.** Delta (Z-order, liquid clustering) reads through `DeltaParquetFileFormat`, a subclass that the exact-class check excludes. Iceberg (sort orders) goes through DSv2. The conf doc says "V1 Parquet file source", and a Delta user will likely read that as covering them. It may be worth stating the subclass exclusion there, and later offering subclasses an explicit opt-in.
- **The benchmark sits at the favorable end of all of this, and on local storage.**
- The fact table is `spark.range` written as a single file, so it is fully sorted on `k`.
- The "clustered" dimension selects one contiguous key range.
- Row groups are capped at 1,500 and 42,000 rows (~3MB). With the default 128MB the whole ~100MB table would be a single row group, so the clustered cases' 33-of-34 row-group skips would not happen.
- The bloom is sized at 13 bits per item, injection is forced (broadcast off, application-side threshold 0), and it runs on `local[1]` against a local filesystem.
The representative rows are "scattered" with default pages: the join is 1.0–1.4x on the wide table and 0.9–1.0x on the narrow one.
My main worry is object storage, where most of these tables live. There, a plain read fetches a row group's columns in one vectored read. This reader issues two, phase 1 for the keys and phase 2 for the rest, and with no pushed data filter phase 2 also reads the offset indexes first. For a row group where the filter prunes little, that is extra request latency with nothing saved, and a local SSD hides it. So the scattered rows above may look worse there, and the clustered ones may gain less than the byte counts suggest. If you get a chance, a run on object storage would tell us a lot, ideally TPC-DS (or similar) at a realistic scale with default write settings and default injection thresholds, noting which queries get `StorageFilters` and how they change.
**Survivor buffer memory**
The survivor buffer (up to 64MB per reader, so N times that per executor) sits outside any `MemoryConsumer`. Registering it would let an allocation be denied, and `spill()` could simply release the buffer and fall back to re-reading the keys.
**Comments**
About 1,000 of the ~2,400 added non-test lines are comments, much of it narrative (alternatives considered, past measurements, the same point restated across `SQLConf`, `ParquetStorageFilter` and the reader). A cleanup that keeps only the invariants and non-obvious trade-offs would make the reader much easier to maintain.
**The public conf doc**
`spark.sql.parquet.storageFilterPushdown.enabled` is user-facing, but its doc is ~25 lines of internals: phases, reading key columns twice, when the reader gives up. I'd keep only what changes a user's decision or answer. Something like:
> If true, the vectorized reader of the V1 Parquet data source may use a runtime filter on join keys, such as a runtime bloom filter, to skip reading the pages and row groups of other columns that hold no matching rows. The filter is still applied after the scan, so results do not change, with two exceptions: with spark.sql.files.ignoreCorruptFiles=true the rows returned from a corrupt file can differ, and a file with an incorrect Parquet page index can return wrong results (set parquet.filter.columnindex.enabled=false to stop relying on it). Only ParquetFileFormat itself is supported, not its subclasses. It can be slower when matching rows are spread across most pages. Matching key values are buffered per row group, up to spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes, outside Spark's memory manager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown via splicing [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5717532541
Opening this as a draft because it depends on #58120, which upgrades Parquet to 1.18.1. The feature needs two APIs that arrive with that upgrade, `RowRanges.Builder` and the public `ParquetFileReader.getRowRanges(int)`.
This branch is stacked on #58120's head, so that PR's commit shows up in the diff here until it merges. Only the second commit is mine. I will rebase onto master and take this out of draft once #58120 is in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5919895966
@dongjoon-hyun thank you. Round 7 is answered in the threads, and the description is up to date.
- A kill or a fatal error now goes out wrapped in a checked `SparkException`, so `ignoreCorruptFiles` cannot read it as a corrupt file. A survivor vector that cannot grow gives splicing up instead of reaching that path.
- The reader checks for a kill per row group and per chunk of phase 1.
- The conf doc and the description say again that a read with no pushed data filter trusts the page index only with this feature.
- Everything else is fixed as you suggested. Each new behavior but the buffer's own catch has a test that fails without it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4158023951
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2823 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading the
+ // rest of a file without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender("partial-read warning")
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val warned = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering"))
+ (result.get, warned)
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString),
+ useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(195L))
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(0L))),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = ThrowsCheckedOn(BoundReference(0, LongType, nullable = false), bad = 7L)
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. A fatal error an expression wrapped is found in the
+ // cause chain, which is where the executor looks for one. Either goes out wrapped in a checked
+ // exception, since `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def readWith(key: Expression): SparkException = intercept[SparkException] {
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ readAll(path, filter)._2.close()
+ }
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException = {
+ TaskContext.setTaskContext(TaskContext.empty())
+ try readWith(key) finally TaskContext.unset()
+ }
+ assertPropagates(killedWhile(ThrowsCheckedOn(bound, bad = 7L, killsTask = true)),
+ classOf[SparkException])
+ assertPropagates(
+ killedWhile(ThrowsCheckedOn(bound, bad = 7L, checked = false, killsTask = true)),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, which is how `WritableColumnVector.reserve`
+ // reports one.
+ val fatal = readWith(ThrowsCheckedOn(bound, bad = 7L, fatalCause = true, checked = false))
+ assertPropagates(fatal, classOf[IllegalStateException])
+ assert(fatal.getCause.getCause.isInstanceOf[OutOfMemoryError],
+ s"the fatal error must still be in the chain for the executor; got ${fatal.getCause}")
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(MarksKilledOn(bound, at = 5L), Literal(50L))), kvSchema())
+ TaskContext.setTaskContext(TaskContext.empty())
+ val thrown = try {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ } finally {
+ TaskContext.unset()
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. Phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
+ val pushed = scanOf(df.queryExecution.executedPlan).simpleString(200)
+ assert(pushed.contains("GreaterThan(v,"),
+ s"the data filter must actually be pushed for this test to mean anything: $pushed")
+ // Storage filter on `k` (key column).
+ val withSF = scanWithStorageFilter(df, threshold = 100L)
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ // Every row satisfies v > 'v_000', so the storage filter alone decides the result.
+ val expected = rows.filter(_._1 >= 100L)
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty)
+ .getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again, same rows, more bytes") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Two things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. And
+ // the fallback must actually have happened, which is observable. It reads the key column a
+ // second time, so it transfers strictly more bytes. Without that second assertion the test
+ // would pass even if the cap never reached the reader.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up past the cap still
+ // has to emit in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = kvSchema()
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") {
+ readCounting(path, formatReader(fileSchema, Nil, storageFilters, countingHadoopConf()))
+ }
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the
+ // row ranges. `k >= 350` keeps a contiguous run, so there is one range to hold.
+ val (plainRows, plainBytes) = run("100b")
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity = 16)
+ try {
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected a row group with no survivor at all; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group past the cap charges its second key read against the byte metric") {
+ // Giving splicing up means phase 2 reads the key columns a second time, while the baseline
+ // counts them once, in phase 1. That extra read is a cost against the saving, so the same file
+ // and filter must report less avoided than they do while splicing. Without that term the metric
+ // would credit the feature with bytes it did transfer.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = kvSchema()
+
+ def avoidedBytes(cap: Long, threshold: Long): Long = {
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))),
+ fileSchema, m, cap)
+ val (_, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => b.column(0).getLong(i), capacity = 16)
+ try m.bytesAvoidedByPageFiltering.value finally reader.close()
+ }
+
+ val spliced = avoidedBytes(64L * 1024 * 1024, 350L)
+ val plain = avoidedBytes(100L, 350L)
+ assert(spliced > 0, s"page filtering has to avoid something here; got $spliced")
+ assert(plain < spliced,
+ s"the second key read must count against the saving; plain=$plain spliced=$spliced")
+
+ // The two halves of the budget are weighed together, and this is the case that says so. Eight
+ // survivors of a long key buffer 8 * (1 + 8) = 72 bytes and fall in one range of 40, so
+ // neither half reaches the 100-byte cap on its own while the sum passes it at the seventh.
+ // Weighing them separately would keep splicing here, and report the larger saving for it.
+ val eightSpliced = avoidedBytes(64L * 1024 * 1024, 393L)
+ val eightPlain = avoidedBytes(100L, 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+ }
+ }
+
+ test("an accumulator sized up front is charged for its byte child when it is sized") {
+ // Every accumulator of a row group after the first sizes a variable-length key's byte child to
+ // what the one before it held, before any value arrives. Charged only as values arrive, the
+ // last accumulator of a row group would hold a whole accumulator's worth of memory for a few
+ // survivors' charge. Six survivors of a 1000-byte key over a capacity of 4 are charged about
+ // 24KB value by value, and about 32KB once the second accumulator's byte child is charged when
+ // it is sized. A cap between the two gives splicing up only in the second case, which the
+ // second read of the key column shows.
+ withTempDir { dir =>
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), REPEAT('x', 996)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val fileSchema = kvSchema(keyType = StringType)
+ val storageFilters = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "4") {
+ readCounting(path, formatReader(fileSchema, Nil, storageFilters, countingHadoopConf()))
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ val (cappedRows, cappedBytes) = run("28000b")
+ assert(splicedRows == 6 && cappedRows == 6,
+ s"the filter keeps keys 15..20 either way; got $splicedRows and $cappedRows")
+ assert(cappedBytes > splicedBytes,
+ "the second accumulator's byte child must count against the cap, so the row group gives " +
+ s"splicing up and reads the key column again; capped=$cappedBytes spliced=$splicedBytes")
+
+ // And the keys spliced from the accumulator sized up front are the right ones.
+ val (pairs, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(storageFilters, fileSchema),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString),
+ capacity = 4)
+ try {
+ assert(pairs == (15 to 20).map(id => (f"$id%04d" + "x" * 996, id.toString)),
+ s"every survivor must come back with its own key; got ${pairs.map(_._2)}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index does not cost the file its page filtering") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, and the
+ // reader answers that by no longer narrowing a row group for the rest of the split. The pushed
+ // data filter's page pruning survives that, so the rows would not show it. The WARN the
+ // exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(50L)))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(Array.fill(length)(0xFF.toByte))
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an Error instead, as hadoop-lzo does, fails before the row group's first batch.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ val outDir = new File(dir, "damaged").getAbsolutePath
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ rows.toDF("k", "v").repartition(1).write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 64L * 1024 * 1024)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .option(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT, 100L)
+ .parquet(outDir)
+ val path = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ .head.getAbsolutePath
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(false, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(false, 16, storageFilter)
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ (read.toSeq, error)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError) = readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry leaves the reader unable to narrow the rest of this file, since the next row
+ // group's store may be as unreadable as this one's. What it must not give up is the pushed data
+ // filter's own page pruning, which parquet did before the reader saw the block, and which a
+ // plain read of the same query keeps. Dropping that would read more than having no storage
+ // filter at all.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warned) = withPartialReadWarning(read(evenKeys))
+ assert(warned, "the read must have discovered the damaged index, or this test proves nothing")
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry has latched.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the survivor queue and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = kvSchema()
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString), capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // Each survivor costs 9 bytes of long key and 40 of its own range. The sum passes the cap of
+ // 1024 at survivor 21, which releases the buffer, and the ranges alone at survivor 26, which
+ // gives the filter up. The capacity decides what the buffer holds when it is released. At
+ // 4096 that is one partly filled set of accumulators, and at 16 a full set in the queue as
+ // well. Both have to end with the queue empty and the filter given up. Keeping one without
+ // the other would leave phase 2 reading non-key columns while emit spliced keys from a queue
+ // holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString))
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSeq == Seq("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does, since the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers conjuncts under the relation's own column names") {
+ // The analyzer spells a reference the way the query did, so under case-insensitive resolution
+ // a bloom can name `_TMP_METADATA_ROW_INDEX` while the relation, and the schema the reader
+ // requests, call the column `_tmp_metadata_row_index`. The planner normalizes what it offers to
+ // the relation's names, so the format's gate sees the name the reader will see and declines.
+ // Without that the gate passed it, `ParquetStorageFilter.create` rejected it on the driver, and
+ // a query that runs with the conf off failed with it on.
+ withTempDir { dir =>
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", s"id AS $rowIndexName", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", rowIndexName, "v")
+ .queryExecution.optimizedPlan
+ def column(name: String): Attribute =
+ relation.output.find(_.name == name).getOrElse(fail(s"no $name in the relation output"))
+
+ def offered(key: Attribute): Seq[Expression] = {
+ val logical = LogicalFilter(
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ scanOf(physical).storageFilters
+ }
+
+ // As a control, an ordinary key spelled in another case is offered, under its own name.
+ val upperK = offered(column("k").withName("K"))
+ assert(upperK.size == 1, s"a bloom over `K` must reach the scan; got $upperK")
+ assert(upperK.head.references.map(_.name).toSet == Set("k"),
+ s"and it must carry the relation's name for the column; got ${upperK.head}")
+
+ val upperRowIndex =
+ offered(column(rowIndexName).withName(rowIndexName.toUpperCase(Locale.ROOT)))
+ assert(upperRowIndex.isEmpty,
+ s"a bloom over the row-index column must not be offered in any case; got $upperRowIndex")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for. A dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`, since `/` is floating-point division in Spark SQL, which would hand every
+ // key expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize, dictionary = dictionary)
+ }
+
+ // Every column chunk of the file, from its footer. The facts the tests below assert about a
+ // fixture are read off these rather than assumed from the write options.
+ private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] =
+ footerOf(filePath).getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq
+
+ private def footerOf(filePath: String): ParquetMetadata = {
+ val conf = spark.sessionState.newHadoopConf()
+ ParquetFooterReader.readFooter(
+ HadoopInputFile.fromPath(new Path(filePath), conf), ParquetMetadataConverter.NO_FILTER)
+ }
+
+ // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String, column: String = "k"): Set[Encoding] =
+ footerChunks(filePath).filter(_.getPath.toDotString == column)
+ .flatMap(_.getEncodings.asScala).toSet
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // builds the plain vectorized reader rather than the late-materialization one. Every read helper
+ // in this suite goes through here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need. The caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ if (storageFilter == null) {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } else {
+ new LateMaterializationParquetRecordReader(useOffHeap, capacity, storageFilter)
+ }
+ } { reader =>
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // A key value as a string in its *internal* representation, so the splicing path and the plain
+ // path can be compared without knowing the type and without external type conversion.
+ // `InternalRow.getAccessor` does the per-type read, which is why this takes the batch's row
+ // rather than the column vector.
+ private def renderValue(row: InternalRow, ordinal: Int, dt: DataType): String =
+ InternalRow.getAccessor(dt)(row, ordinal) match {
+ case null => "null"
+ // Two equal arrays are two objects, so the default toString would compare identities.
+ case bytes: Array[Byte] => bytes.mkString(",")
+ case value => value.toString
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path. Whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.getRow(i), 0, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage, one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why. There is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ } {
+ val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded"
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // This is a regression test. TimeType is an AtomicType that passes every planning-time gate
+ // (it is batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a
+ // TIME join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // What FileSourceStrategy offers, what ParquetStorageFilter.create admits and what the reader
+ // can copy are one list, because isSupportedKeyType answers from ValueCopier. Both are asserted
+ // here. A type the copier has no case for would be a task failure rather than a planning-time
+ // rejection, and the delegation is what rules that out.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ assert(ValueCopier.forType(dt) != null, s"$dt should have a value copier")
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ intercept[IllegalStateException](ValueCopier.forType(dt))
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, which fixes the key row's
+ // layout. The reader pairs each key column's batch slot with its key-row position, so nothing
+ // depends on that order. Every other fixture mentions its keys in column order, and production
+ // does not promise that, since the conjuncts arrive in `afterScanFilters` order, which follows
+ // the join. So this has two key columns mentioned the other way round, and asserts both the
+ // sorted layout and that every key still comes back with its own row.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // The emit path rewrites the key slots alone, and a partition slot is not one of them. It sits
+ // past the data columns and must keep the constant vector `initBatch` populated for it.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite. OffHeapColumnVector.close() frees
+ // the native buffer, so a read after close is a crash rather than stale data. A second close
+ // is harmless, since close() zeroes the addresses it freed. capacity = 16 over 71 survivors
+ // forces 5 emits, each closing the previous emit's dequeued key vectors. The other 29 rows
+ // are what makes a declining reader fail this.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(30L))
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (30L to 100L),
+ s"expected the 71 surviving keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // One ColumnarBatch is handed out for the whole read and its key slots are rewritten per
+ // batch, so a consumer holding on to an earlier getCurrentValue() would read the wrong
+ // vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new LateMaterializationParquetRecordReader(false, 8, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution, a column missing from the physical file -----
+
+ test("non-key column missing from a file: the evolved table reads and credits avoided bytes") {
+ // Schema evolution end to end, with the non-key column rather than the key missing from the
+ // older file. The clipped requested schema keeps the missing column, which the byte walks skip
+ // by themselves, since they go over the block's chunks and the older file has no chunk for it.
+ // What this pins is that such a read returns the right rows and still credits a saving.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // The older file has (k, v), the newer one (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSeq.sorted
+ val expected = (150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // A positive total says the byte walks ran over the older file's row groups as well.
+ // Asserting non-negativity would prove nothing. `SQLMetric.add` drops a negative and the
+ // value cannot go below zero.
+ val bytesRg = withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value + bytesPf.value > 0,
+ s"the walk must credit some avoided bytes; got rg=${bytesRg.value} pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = scanOf(spark.read.parquet(path).select("k", "v").queryExecution.executedPlan)
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Seq[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read. The rows come
+ // back sorted, so a row returned twice shows.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Seq[(Long, Option[Long])]) = {
+ val withSF = withStorageFilters(spark.sql(s"SELECT id, k FROM $table")) { attr =>
+ Seq(buildFilter(attr("k")))
+ }
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSeq.sorted
+ (withSF, collected)
+ }
+
+ test("all keys missing: an error on the constant reads the file rather than skipping it") {
+ // A file with none of the filter's key columns makes the predicate constant over it, so the
+ // reader decides once, from the values `initBatch` materialized for those columns. That
+ // evaluation can throw exactly as a row's value can. Here the older file's `k` reads as its
+ // DEFAULT, which is not a number, and the cast fails under ANSI. Failing open reads the file
+ // the way a plain scan would, so its rows survive. Skipping it would lose them, and rethrowing
+ // would fail a query a plain read answers.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true",
+ SQLConf.ANSI_ENABLED.key -> "true") {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql("ALTER TABLE evolved ADD COLUMN k STRING DEFAULT 'not-a-number'")
+ spark.sql("INSERT INTO evolved VALUES (4, '40'), (5, '50')")
+ val withSF = withStorageFilters(spark.sql("SELECT id, k FROM evolved")) { attr =>
+ Seq(GreaterThanOrEqual(Cast(attr("k"), LongType), Literal(1L)))
+ }
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val ids = collectRows(withSF).map(_.getLong(idPos)).toSeq.sorted
+ assert(ids == Seq(1L, 2L, 3L, 4L, 5L),
+ s"the older file must fail open and the newer one pass the cast; got $ids")
+ }
+ }
+ }
+
+ test("all keys missing: a checked exception on the constant reads the file") {
+ // The constant evaluation fails open under the same rule as a row's, a checked exception
+ // included. The older file's `k` reads as its DEFAULT 7, which the key expression throws on,
+ // so that file must be read the way a plain scan would. The newer file's keys evaluate
+ // normally and are rejected, since no Filter runs above this scan.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(ThrowsCheckedOn(k, bad = 7L), Literal(100L)))
+ assert(collected == Seq((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L))),
+ s"the older file must fail open and the newer one be filtered; got $collected")
+ }
+ }
+ }
+
+ test("a file with none of the projected non-key columns is read plainly") {
+ // The clipped requested schema keeps a column the file does not have, so a projected non-key
+ // column added after the older file was written still looks like something for phase 2 to
+ // read. Phase 2 reads nothing for it, so for that file the filter could only add cost, and the
+ // reader declines it there. With no Filter above this scan, the older file's rows come back
+ // unfiltered while the newer file's are still filtered.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTable("evolved_value") {
+ spark.sql("CREATE TABLE evolved_value (k BIGINT, v STRING) USING parquet")
+ spark.sql("INSERT INTO evolved_value VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+ spark.sql("ALTER TABLE evolved_value ADD COLUMN w STRING")
+ spark.sql("INSERT INTO evolved_value VALUES (4, 'd', 'x'), (5, 'e', 'y')")
+ val withSF =
+ scanWithStorageFilter(spark.sql("SELECT k, w FROM evolved_value"), threshold = 5L)
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val keys = collectRows(withSF).map(_.getLong(kPos)).toSeq.sorted
+ assert(keys == Seq(1L, 2L, 3L, 5L),
+ s"the older file must be read unfiltered and the newer one filtered; got $keys")
+ }
+ }
+ }
+
+ test("a file with none of a projected struct's requested fields is read plainly") {
+ // Under the legacy returnNullStructIfAllFieldsMissing, the clipped schema keeps a struct none
+ // of whose requested fields the file has, and adds no field it has, so phase 2 would read
+ // nothing for it. Only the struct's leaves are missing columns, not the struct, so the reader
+ // asks the file schema about every non-key leaf. The filter rejects every row, so a declined
+ // file comes back whole and one that is not comes back empty.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 20).selectExpr("id AS k", "named_struct('a', CAST(id AS INT)) AS s"),
+ rowGroupSize = 64 * 1024L)
+ val schema =
+ new StructType().add("k", LongType).add("s", new StructType().add("b", IntegerType))
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(1000L)))
+ withSQLConf(SQLConf.LEGACY_PARQUET_RETURN_NULL_STRUCT_IF_ALL_FIELDS_MISSING.key -> "true") {
+ val keys = readLongs(formatReader(schema, Nil, storageFilters), path, ordinal = 0)
+ assert(keys == (0L until 20L), s"the file must be read plainly; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Seq[(Long, Option[Long])] =
+ Seq((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test. The default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Seq[(Long, Option[Long])] = Seq((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected, s"got $collected; expected $expected")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(StorageFilterMetrics.ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Seq((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got $rejectsNull")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Seq[(Long, Option[Long])] =
+ Seq((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got $acceptsNull")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ val collected = collectRows(withSF).map(r => (r.getLong(0), r.getLong(2))).toSeq.sorted
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1))
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got ${collected.take(5)}")
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF =
+ scanWithStorageFilter(spark.read.parquet(outDir).select("k", "s"), threshold = 195L)
+ val collected = collectRows(withSF)
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSeq.sorted
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k"))
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ test("a storage filter binds to the right column with partition and metadata columns in output") {
+ // `preparedStorageFilters` binds the filter's references against `output.take(requiredSchema
+ // .length)`, and `output` is the data columns, then the generated metadata columns, then the
+ // partition columns, then the constant metadata ones. A projection carrying all four is where a
+ // wrong prefix would bind the predicate to another column and drop rows the filter keeps, which
+ // the post-scan Filter cannot put back. The file's first column `a` is not projected, so the
+ // key sits at a different position in the relation's data schema than in the scan's.
+ withTempDir { dir =>
+ val outDir = new File(dir, "parted").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id * 7 AS a", "id AS k", "CONCAT('v_', id) AS v", "CAST(id % 3 AS INT) AS p")
+ .repartition(1)
+ .write
+ .partitionBy("p")
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select(
+ col("k"), col("v"), col("p"),
+ col("_metadata.row_index").as("ri"), col("_metadata.file_size").as("fs"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ // The assumption the binding rests on, stated where a change to `output` would break it.
+ assert(withSF.output.take(withSF.requiredSchema.length).map(_.name) ==
+ withSF.requiredSchema.fieldNames.toSeq,
+ s"the scan's leading attributes must be its data columns; got " +
+ s"${withSF.output.map(_.name)}")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val pPos = withSF.output.indexWhere(_.name == "p")
+ def collectAll(plan: SparkPlan): Seq[Seq[Any]] =
+ collectRows(plan).map(_.toSeq(plan.output.map(_.dataType))).toSeq
+ .sortBy(_(kPos).asInstanceOf[Long])
+ val filtered = collectAll(withSF)
+ // The same scan without the filter is the oracle, so the row indexes and file sizes come
+ // from the read rather than from arithmetic over how the partitions were written.
+ val expected = collectAll(withSF.copy(storageFilters = Nil))
+ .filter(_(kPos).asInstanceOf[Long] >= 150L)
+ assert(filtered == expected,
+ s"expected ${expected.size} rows with their own partition and metadata values; " +
+ s"got ${filtered.size}")
+ assert(filtered.map(_(pPos)).distinct.size == 3,
+ s"every partition value must still be represented; got ${filtered.map(_(pPos)).distinct}")
+ }
+ }
+ }
+
+ // ----- Planner gates -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false. It may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ //
+ // This hook answers about shapes and types alone. Whether the conjunct is deterministic and
+ // whether its references are projected columns of this scan are plan-visible, so they stay with
+ // the planner; `FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter`
+ // covers that half.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ // A cast key is supported. In ANSI mode it can throw on a row an earlier conjunct would have
+ // dropped, and the reader handles that where it arises, by giving the filter up for the row
+ // group. Declining here instead would also decline every widening cast, which is what type
+ // coercion inserts for a join between an int and a bigint column.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+
+ // The conf is read in the format rather than in the planner, and it is the per-scan question
+ // that carries it, asked once before anything per conjunct.
+ assert(!format.supportsStorageFilterPushdown(spark), "the feature is off with the conf off")
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ assert(format.supportsStorageFilterPushdown(spark), "and on with it on")
+ assert(!subclass.supportsStorageFilterPushdown(spark), "a subclass must not claim support")
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ assert(format.supportsStorageFilter(onCast),
+ "a cast key is pushed, and an evaluation error gives the row group up")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The rows are right either way, since the bloom stays in the post-scan Filter. What this gate
+ // saves is offering the scan a filter its reader cannot apply.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one. Every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSeq.sorted
+ assert(keys == (1L to 50L),
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support declines rather than delegating") {
+ // The default `FileFormat.buildReaderWithStorageFilters` answers None, and the caller falls
+ // back to the ordinary builder. That is what keeps the two builders from being able to call
+ // each other, and declining is safe because the post-scan Filter still holds the conjunct.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(1L)))
+ val declined = new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration(), Map.empty)
+ assert(declined.isEmpty, "the default must not build a reader of its own")
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilterPushdown(spark),
+ "and it must not claim support either")
+ }
+
+ test("a scan whose format declines falls back to the ordinary reader and returns every row") {
+ // The check above is that the default answers None. This is the other half, that `inputRDD`
+ // then builds the ordinary reader and the scan still produces its rows. A `ParquetFileFormat`
+ // subclass is the shape that reaches it, since it inherits the override and declines by class,
+ // and its ordinary builder is the one an extension would have customized.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, threshold = 195L)
+ val declining = withSF.copy(
+ relation = withSF.relation.copy(fileFormat = new ParquetFileFormat() {})(spark))
+ val collected = executePlanCollect(declining).toSeq.sorted
+ assert(collected == rows,
+ s"a declining format reads the file plainly; got ${collected.size} of ${rows.size} rows")
+ }
+ }
+ }
+
+ Seq(false, true).foreach { aqe =>
+ test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") {
+ // AQE is on by default in production, and it is where the bloom subquery is planned by
+ // PlanAdaptiveSubqueries rather than PlanSubqueries, which is the path
+ // preparedStorageFilters' ScalarSubquery materialization depends on.
+ withBloomFilterTables {
+ val baseConf = Map(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString)
+ def run(pushdown: Boolean): (Int, Seq[(Long, Long)]) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, rows) = runBloomFilterJoin()
+ (countBloomFiltersInStorageFilters(plan),
+ rows.map(r => (r.getLong(0), r.getLong(1))).toSeq.sorted)
+ }
+ val (offFilters, off) = run(false)
+ val (onFilters, on) = run(true)
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ // Equal results are guaranteed by the post-scan Filter whether or not the scan applied the
+ // filter, so they alone would not notice AQE dropping it on the way to the final plan.
+ assert(offFilters == 0 && onFilters == 1,
Review Comment:
Added. The arm sums the two row-exclusion metrics of the final plan's scan that carries the filter, and asserts they are positive.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2823 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading the
+ // rest of a file without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender("partial-read warning")
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val warned = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering"))
+ (result.get, warned)
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString),
+ useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(195L))
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(0L))),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = ThrowsCheckedOn(BoundReference(0, LongType, nullable = false), bad = 7L)
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. A fatal error an expression wrapped is found in the
+ // cause chain, which is where the executor looks for one. Either goes out wrapped in a checked
+ // exception, since `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def readWith(key: Expression): SparkException = intercept[SparkException] {
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ readAll(path, filter)._2.close()
+ }
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException = {
+ TaskContext.setTaskContext(TaskContext.empty())
+ try readWith(key) finally TaskContext.unset()
+ }
+ assertPropagates(killedWhile(ThrowsCheckedOn(bound, bad = 7L, killsTask = true)),
+ classOf[SparkException])
+ assertPropagates(
+ killedWhile(ThrowsCheckedOn(bound, bad = 7L, checked = false, killsTask = true)),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, which is how `WritableColumnVector.reserve`
+ // reports one.
+ val fatal = readWith(ThrowsCheckedOn(bound, bad = 7L, fatalCause = true, checked = false))
+ assertPropagates(fatal, classOf[IllegalStateException])
+ assert(fatal.getCause.getCause.isInstanceOf[OutOfMemoryError],
+ s"the fatal error must still be in the chain for the executor; got ${fatal.getCause}")
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(MarksKilledOn(bound, at = 5L), Literal(50L))), kvSchema())
+ TaskContext.setTaskContext(TaskContext.empty())
+ val thrown = try {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ } finally {
+ TaskContext.unset()
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. Phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
Review Comment:
Rewritten as you suggest. The pushed filter drops the first of four 100-row pages, and the storage filter keeps keys up to 250, so each keeps rows the other drops. The test asserts the pushed filter's narrowing, the exact rows both keep, and the 150 rows excluded inside the row group.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
Review Comment:
Both rewritten. The queue check's comment says that without it the key slots would point at the persistent vectors, which a spliced row group resets but never reads into, so the batch would carry stale key values as real ones. `readerRangesFor` says its null is required, since a whole-block read hands back a store with no row indexes and `forRead` refuses ranges then.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
Review Comment:
Added to the class javadoc. A caller must fetch `batch.column(i)` again after every `nextBatch()`.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:
##########
@@ -190,6 +191,64 @@ class ParquetFileFormat
filters: Seq[Filter],
options: Map[String, String],
hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow] = {
+ buildParquetReader(
+ sparkSession, dataSchema, partitionSchema, requiredSchema, filters, Nil, options, hadoopConf,
+ Map.empty)
+ }
+
+ /**
+ * The conf that turns this on is read here rather than in the planner, the way `supportBatch`
+ * reads its own confs, so a Parquet-named conf does not decide for a format that is not Parquet.
+ *
+ * Subclasses answer false on purpose, even though they inherit this reader, since a subclass may
+ * customize reading by overriding `buildReaderWithPartitionValues`, and a scan with storage
+ * filters routes through `buildReaderWithStorageFilters` instead, which would silently bypass
+ * whatever the subclass does.
+ */
+ override def supportsStorageFilterPushdown(sparkSession: SparkSession): Boolean =
+ getSqlConf(sparkSession).parquetStorageFilterPushdownEnabled &&
+ getClass == classOf[ParquetFileFormat]
+
+ override def supportsStorageFilter(expr: Expression): Boolean =
+ ParquetStorageFilter.isSupportedStorageFilter(expr)
+
+ /** See `StorageFilterMetrics` for what each of these counts. */
+ override def storageFilterMetrics(sparkContext: SparkContext): Map[String, SQLMetric] =
+ StorageFilterMetrics.create(sparkContext).toMap
+
+ override def buildReaderWithStorageFilters(
+ sparkSession: SparkSession,
+ dataSchema: StructType,
+ partitionSchema: StructType,
+ requiredSchema: StructType,
+ filters: Seq[Filter],
+ storageFilters: Seq[Expression],
+ options: Map[String, String],
+ hadoopConf: Configuration,
+ storageFilterMetrics: Map[String, SQLMetric])
+ : Option[PartitionedFile => Iterator[InternalRow]] = {
+ // The same subclass exclusion as `supportsStorageFilterPushdown`, asked again because this
+ // entry point is reachable without the planner having asked it, and `None` sends the caller to
+ // the ordinary builder, which is the one a subclass overrides.
+ if (getClass != classOf[ParquetFileFormat]) {
Review Comment:
Both now go through one private `isExactlyParquetFileFormat`. It is private on purpose, so a subclass cannot opt in by overriding one entry point and still be declined by the other. Subclasses are not meant to opt in, since one may customize `buildReaderWithPartitionValues`, which a storage-filter scan bypasses. The second check is the one you asked for in round 4 (https://github.com/apache/spark/pull/58895#discussion_r4108708266).
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1013 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #reportOnce} has already reported. */
+ private boolean reportedGivingUp;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, the storage filter no
+ * longer narrows a row group for the rest of this split. The pushed data filter's own page
+ * pruning still applies, since parquet builds those ranges and hands back whole any block it has
+ * no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ giveUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
Review Comment:
Added a note at the catch. It names `MissingOffsetIndexException`, and the `ColumnIndexStore` and `OffsetIndex` that `ParquetBlockAccounting` reads, as parquet's internal API to revisit on an upgrade. It also points at the damaged-index test, which fails if the catch stops matching. Keeping both uses in one file would mean moving the phase-2 read out of the reader, so they stay where they are.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java:
##########
@@ -106,6 +107,7 @@ public VectorizedColumnReader(
ColumnDescriptor descriptor,
boolean isRequired,
PageReadStore pageReadStore,
+ RowRanges rowRanges,
Review Comment:
Added. The 9-argument constructor is back as an overload that passes null ranges.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java:
##########
@@ -26,23 +27,20 @@
/**
* Helper class to store intermediate state while reading a Parquet column chunk.
+ *
+ * <p>There is one subclass per way of saying which rows to include, which is every row of the
+ * chunk, the ranges the caller of the read already held, or the one row index per row that a page
+ * store describes parquet's own filtering with. {@link #forRead} picks between them. Everything
+ * else about a read is the same whichever it is, and lives here.
*/
-final class ParquetReadState {
- /** A special row range used when there is no row indexes (hence all rows must be included) */
- private static final RowRange MAX_ROW_RANGE = new RowRange(Long.MIN_VALUE, Long.MAX_VALUE);
-
+abstract class ParquetReadState {
Review Comment:
Kept as it is. The row-index route is master's code on the plain read path, and it is master's eager version on purpose, since the lazy one measured 2% to 8% slower in the same-machine A/B from round 6 (https://github.com/apache/spark/pull/58895#discussion_r4137343688). Building the plain path's ranges through `RowRanges.builder()` would change that path again and need the same measurement. I'd rather leave the plain read path alone in this PR.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -179,13 +184,13 @@ public VectorizedParquetRecordReader(boolean useOffHeap, int capacity) {
}
/**
- * Implementation of RecordReader API.
+ * Implementation of RecordReader API. Super's delegates to the overload below, which runs
+ * {@link #initializeInternal()}, so this does not run it a second time.
*/
@Override
public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptContext)
Review Comment:
This one has to stay. Removing it in round 5 tripped MiMa with a `DirectAbstractMethodProblem` on `org.apache.hadoop.mapreduce.RecordReader.initialize`, which the `sql.execution.*` exclude does not cover. See the correction in https://github.com/apache/spark/pull/58895#discussion_r4135661837. The override only calls `super`, and the test still covers the single initialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4189021628
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1084 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A byte child sized up front is
+ * charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}. A
+ * fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // decoders of their last page.
+ closeDataPages();
+ releaseRowGroupReaders();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (reportedMessages.add(OFFSET_INDEX_UNREADABLE)) {
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn(OFFSET_INDEX_UNREADABLE, MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long survivors = 0L;
+ long firstSurvivorAt = 0L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_DECODING, true);
+ // The failed decode may have grown the scratch vectors, and phase 2 reads the same pages
+ // into vectors of its own. So capacity-sized ones replace them, and beside a plain read's
+ // own vectors the reader then holds one capacity-sized batch of key vectors.
+ closeAll(keyScratchVectors);
+ allocateKeyScratch();
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = predicate.eval(keyScratchBatch.getRow(r));
+ } catch (Throwable t) {
+ // Fall back on anything but a task kill or a fatal error. The predicate runs here without
+ // the conjuncts before it in the plan, so it can throw on a row a plain scan never
+ // evaluates it on (see `ParquetStorageFilter.isSupportedStorageFilter`). Giving the
+ // filter up returns every row the pushed filter kept, and the post-scan Filter decides
+ // them in its own order, raising the same error for a row it does evaluate.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK exception
+ // (`timestamp_seconds` of a decimal) or an `AssertionError`, and a Hive UDF throws a
+ // checked SparkException. Rethrowing would fail a query a plain read answers, or drop
+ // the rest of the file silently under `ignoreCorruptFiles`.
+ //
+ // A fatal error goes out even on a row an earlier conjunct of the plan would have
+ // rejected, so the reader never runs on after one the executor would take as fatal.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_EVALUATION, false);
+ return null;
+ }
+ if (!survives) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ long rowsBefore = pushedFilterRowCount - remaining + r;
+ if (survivors++ == 0) firstSurvivorAt = rowsBefore;
+ if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ if (currentKeyAccumulators == null) {
+ // The survivors from this one on, at the rate seen since the first one, so that rows
+ // before it, which a clustered key leaves, do not dilute the rate.
+ long expected = (long) Math.ceil(
+ (double) survivors * (remaining - r) / (rowsBefore - firstSurvivorAt + 1));
+ splicedBytes += allocateKeyAccumulators(cap - splicedBytes - rangeBytes, expected);
+ }
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // The buffer is this reader's own and optional, so failing to fill it gives splicing up
+ // rather than the filter, and the memory goes back at once. What fails here:
+ // - a survivor vector that cannot grow;
+ // - a key value the predicate did not read on this row, such as `b` in
+ // `coalesce(a, b)`, since the copy decodes every key;
+ // - a bug of ours, which only the WARN then shows.
+ // Phase 2 reads the key columns the way a plain read does, so a value that cannot be
+ // decoded fails the read only where a plain read fails.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together.
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, and returns the charge for the
+ * byte children it sized up front. The rest of the set is charged per row as survivors arrive.
+ * A variable-length key's byte child is reserved for what the set queued before it held, scaled
+ * to the {@code expectedRows} survivors this set can expect, where that is more than its default
+ * allocation. So it does not grow one doubling at a time for every capacity's worth of
+ * survivors, and a sparse last set is not sized as a full one. With its default doubling,
Review Comment:
**The rate I suggested in R10-1 cannot see ahead, so this sentence and the comment at L810-811 overstate it.** Within a run of consecutive survivors, the rate since the first survivor is 1, so a set that opens inside a clustered run is still sized for the rows left, up to a full set, however soon the run ends. With survivors at rows 400,000 to 409,999 of a 1M-row row group, at capacity 4096 with 1000-byte keys, the third set still reserves 4,096,000 value bytes, charged 16,384,000, for 1,808 values, which are R10-1's own numbers. My formula, counted from the row group's start, would have sized it as a full set too, so "which removes the over-reservation" was wrong of me.
The other way, survivors scattered ahead of a dense run still dilute the rate. With 900 scattered survivors in rows 0 to 119,999 of a 128,000-row row group and every row after that surviving, the second set expects ceil(4097 * 4804 / 123,197) = 160 and receives 4,096 values, so it grows through four doublings, and the third expects 46 and receives 708, where round 10's `remaining - r` sized both exactly.
Neither opens a give-up window, since an over-sized set is always the row group's last one and the check covers its rows. Since no rate can see a run end, could the javadoc say so instead, e.g. "The rate cannot see ahead: a set that opens while every row since the first survivor has survived is sized for the rows left, up to a full set, however soon that run ends, and survivors scattered ahead of a dense run lower the rate its sets are sized at, so those grow", with the comment at L810-811 qualified to match?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3742 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t: VirtualMachineError => fail(s"the reader must fall back on $e; got $t", t)
Review Comment:
Nit: the `fallsBack` I suggested in R10-6 should also catch `LinkageError`, which ScalaTest aborts the suite on as well, since this round passes it an `ExceptionInInitializerError` at L696-697. A regression that rethrows such an error as it is when no fatal error is found within the depth would still pass L693, whose OOM sits at level 3, but would throw the EIIE out of here at L696. Could this be `case t @ (_: VirtualMachineError | _: LinkageError) =>`?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3742 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t: VirtualMachineError => fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the survivors it can expect and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the survivors it can still expect at the rate seen since the row group's
+ // first survivor. That memory is taken before any value arrives, so it is charged then. Six
+ // survivors over a capacity of 4 fill one set of four 1000-byte keys, and the second set starts
+ // two rows before the end of the row group, with every row since the first survivor kept. So
+ // it is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000
+ // bytes each, so they add 8000 more once the sizing is used up, and the row group holds 32142
+ // bytes in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or
+ // a double charge would each get wrong. The second set's byte child is reserved for 2000
+ // bytes, which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000.
+ // And the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ assertSpliceCharge(32142L, rows = 6) { cap =>
+ readUnderCap(path, laterSetSchema, laterSetFilter, cap, batchSize = 4)
+ }
+
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ if (cap >= 32142L) {
+ assert(secondSet == Seq(4000),
+ s"the second set must be sized for the two values left under cap $cap; got $secondSet")
+ }
+ }
+ }
+ }
+
+ test("a later accumulator set is sized only when it fits with its rows' fixed part and ranges") {
+ // The second set's two keys are 500 bytes here, shorter than the first set's 1000, so growing
+ // value by value is the cheaper path: 20142 bytes in all against 24142 sized. A cap of 24222
+ // sizes the set, whose byte child is then reserved for 2000 bytes and holds 4000. One of
+ // 24221 leaves 8113 bytes after the first set and its range, a byte short of the 8000 of
+ // sizing, the 34 of the two rows' fixed part and the 80 of a range for each. So the set grows
+ // instead, to a 1000-byte child, and keeps splicing. A cap of 20141 gives splicing up.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 500)
+ def run(maxSplicedBytes: String): Long =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)._2
+
+ val uncapped = run("64MB")
+ assert(run("24222b") == uncapped, "a cap that fits the sizing must keep splicing")
+ assert(run("24221b") == uncapped, "and so must one that only fits growing value by value")
+ assert(run("20141b") > uncapped, "a cap under both must give splicing up")
+
+ Seq(Long.MaxValue -> 4000, 24222L -> 4000, 24221L -> 1000).foreach { case (cap, child) =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected, s"every survivor must come back with its own key under cap $cap")
+ assert(secondSet == Seq(child),
+ s"under cap $cap the second set's byte child must hold $child bytes; got $secondSet")
+ }
+ }
+ }
+
+ test("a later set is sized only when the ranges of its scattered survivors fit too") {
+ // Keys 9 to 12 fill the first set over a capacity of 4, and keys 14 and 17 come scattered, each
+ // in a range of its own. The second set opens at key 14, sized for the four rows the rate
+ // since the first survivor promises, and its sizing is charged up front. So it is sized only
+ // when the cap holds a range for each of those rows as well. Without that, caps of 32216 to
+ // 32221 sized the set and then passed the cap on key 17's range, giving splicing up where a
+ // smaller cap of 32215, which grows the set instead, kept it. For this file, every cap from
+ // the smallest that splices up must splice, and every cap returns the same six rows.
+ withTempDir { dir =>
+ val (path, _) = writeLaterSetFile(dir, lastTwoBytes = 1000)
+ val prefix = Substring(BoundReference(0, StringType, nullable = true), Literal(1), Literal(4))
+ val keys = Seq("0009", "0010", "0011", "0012", "0014", "0017").map(Literal(_))
+ def readUnder(cap: Long): (Int, Long) =
+ readUnderCap(path, laterSetSchema, Seq(In(prefix, keys)), s"${cap}b", batchSize = 4)
+ val (_, uncapped) = readUnder(64L * 1024 * 1024)
Review Comment:
Nit: this writes out `assertSpliceCharge` by hand, the uncapped read, 24221 giving splicing up and 24222 keeping it, and drops the uncapped read's row count. 24222 is the grown path's charge here, 16068 + 3 * 40 + 8034. Could it be `assertSpliceCharge(24222L, rows = 6)(readUnder)` with `readUnder` taking the cap string, and a loop over 32215, 32216, 32221 and 32222 comparing against `readUnder("64MB")`?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import scala.util.control.NonFatal
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while a code-generated `created`
+ // holds the deserialized filter instead. Dropping the expression lets a columnar scan's task
+ // reclaim those bytes for the rest of its life. An interpreted `created`, the fallback when
+ // codegen fails or is off, keeps the expression itself, so the bytes stay. A row-based scan
+ // keeps them anyway, since its closure captures the scan node, whose storage filters hold the
+ // same bytes.
+ // Safe because every task attempt deserializes its own copy from the driver's bytes, and this
+ // lazy val is the only reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while applying a storage filter, if it must not fall back on
+ * it by giving the filter up. `fromRead` says the error came from decoding the key pages rather
+ * than from evaluating the filter. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped when the error is one the executor's kill case takes,
+ * see [[throwIfKilled]]. The error may be the kill's interrupt surfacing through a UDF that
+ * wrapped it. `PythonRunner` reads a kill off the task context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had a plain plan raised `e`, goes out as itself, so the executor finds
+ * it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in an `UnknownError`, a fatal error of another type.
+ * `FileScanRDD` would take a bare `InternalError` from a reader for a corrupt file under
+ * `ignoreCorruptFiles`. A `NonFatal` wrapper does not work either. With a kill pending, the
+ * executor reports one as killed and keeps running. A plain plan raises it from the post-scan
+ * Filter, where neither happens, and the executor stops. It stops on the `UnknownError` too, at
+ * any `spark.executor.killOnFatalError.depth` of 1 or more, since that is at the top.
+ *
+ * An error from decoding differs in three ways:
+ * - a plain read raises it from the reader, where `FileScanRDD` wraps a `NonFatal` one in
+ * FAILED_READ_FILE before the executor looks. So a fatal error deeper in the chain of a
+ * `NonFatal` one has to sit one level shallower to count, while any other error is searched
+ * as deep as the executor does. Under `ignoreCorruptFiles` a plain read may skip the rest of
+ * the file on it instead.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ *
+ * Anything else falls back. An evaluation error is the filter's own, which the post-scan Filter
+ * raises again for a row it does evaluate. A decoding error is met again by the plain read of
+ * the same pages.
+ */
+ def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new UnknownError("A storage filter met a fatal error it must not absorb")
+ .initCause(internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ val depth = if (fromRead && NonFatal(e)) fatalErrorDepth - 1 else fatalErrorDepth
+ if (Executor.isFatalError(e, depth)) {
+ // The one `Executor.isFatalError` found, which no SparkOutOfMemoryError can precede.
+ val fatal = Iterator.iterate(e)(_.getCause).find(Utils.isFatalError).get
+ if (!corrupt(fatal)) throw out(fatal)
+ }
+ }
+
+ /**
+ * Whether `e` is how `WritableColumnVector.reserve` reports a vector that ran out of memory to
+ * grow, a plain `RuntimeException` caused by an `OutOfMemoryError`. The one it raises with no
+ * cause, for a capacity past what a vector can hold, has no fatal error to find, so the reader
+ * falls back on it anyway.
+ */
+ private def isVectorGrowthFailure(e: Throwable): Boolean =
+ e.getClass == classOf[RuntimeException] && e.getCause.isInstanceOf[OutOfMemoryError]
+
+ /**
+ * Throws a pending task kill, for the reader's loops that can run long without returning to
+ * `FileScanRDD`, which is where a plain read meets one.
+ */
+ def throwIfKilled(): Unit = throwIfKilled(null)
+
+ /**
+ * Throws a pending task kill, wrapped in a checked [[SparkException]] around `surfaced`, the
+ * error the kill surfaced as, if any. Under `ignoreCorruptFiles`, `FileScanRDD` would log any
+ * `RuntimeException` or `IOException` from a reader as a corrupt file and go on to open the
+ * split's remaining files, since it checks for a kill only between batches. The executor reports
+ * the task as killed either way. This covers the kills this reader sees at its own checks and
+ * catches. One that surfaces as an `IOException` from `readFilteredRowGroup` or from phase 2's
+ * decoding reaches that catch as it does for a plain read.
+ *
+ * The executor reports a task as killed only for an `InterruptedException` or a `NonFatal`
+ * error. So a `surfaced` error of any other type goes out as it is, the way a plain read raises
+ * it when decoding meets the same error. For example a codec's `InternalError`, or a native
Review Comment:
Nit, on the PR description: it still says, of a pending kill, "Any other error goes out as it is, the way a plain read raises it", which is the sentence R10-8 narrowed here to errors met while decoding. Its MemoryConsumer follow-up also still says a child is sized up front "only when it grows the child and the charge fits", without the fixed part and the range per row of this round. Could both be updated, since the description becomes the commit message?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import scala.util.control.NonFatal
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while a code-generated `created`
+ // holds the deserialized filter instead. Dropping the expression lets a columnar scan's task
+ // reclaim those bytes for the rest of its life. An interpreted `created`, the fallback when
+ // codegen fails or is off, keeps the expression itself, so the bytes stay. A row-based scan
+ // keeps them anyway, since its closure captures the scan node, whose storage filters hold the
+ // same bytes.
+ // Safe because every task attempt deserializes its own copy from the driver's bytes, and this
+ // lazy val is the only reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while applying a storage filter, if it must not fall back on
+ * it by giving the filter up. `fromRead` says the error came from decoding the key pages rather
+ * than from evaluating the filter. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped when the error is one the executor's kill case takes,
+ * see [[throwIfKilled]]. The error may be the kill's interrupt surfacing through a UDF that
+ * wrapped it. `PythonRunner` reads a kill off the task context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had a plain plan raised `e`, goes out as itself, so the executor finds
+ * it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in an `UnknownError`, a fatal error of another type.
+ * `FileScanRDD` would take a bare `InternalError` from a reader for a corrupt file under
+ * `ignoreCorruptFiles`. A `NonFatal` wrapper does not work either. With a kill pending, the
+ * executor reports one as killed and keeps running. A plain plan raises it from the post-scan
+ * Filter, where neither happens, and the executor stops. It stops on the `UnknownError` too, at
+ * any `spark.executor.killOnFatalError.depth` of 1 or more, since that is at the top.
+ *
+ * An error from decoding differs in three ways:
+ * - a plain read raises it from the reader, where `FileScanRDD` wraps a `NonFatal` one in
+ * FAILED_READ_FILE before the executor looks. So a fatal error deeper in the chain of a
+ * `NonFatal` one has to sit one level shallower to count, while any other error is searched
+ * as deep as the executor does. Under `ignoreCorruptFiles` a plain read may skip the rest of
+ * the file on it instead.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ *
+ * Anything else falls back. An evaluation error is the filter's own, which the post-scan Filter
+ * raises again for a row it does evaluate. A decoding error is met again by the plain read of
+ * the same pages.
+ */
+ def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new UnknownError("A storage filter met a fatal error it must not absorb")
+ .initCause(internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ val depth = if (fromRead && NonFatal(e)) fatalErrorDepth - 1 else fatalErrorDepth
+ if (Executor.isFatalError(e, depth)) {
+ // The one `Executor.isFatalError` found, which no SparkOutOfMemoryError can precede.
+ val fatal = Iterator.iterate(e)(_.getCause).find(Utils.isFatalError).get
+ if (!corrupt(fatal)) throw out(fatal)
+ }
+ }
+
+ /**
+ * Whether `e` is how `WritableColumnVector.reserve` reports a vector that ran out of memory to
+ * grow, a plain `RuntimeException` caused by an `OutOfMemoryError`. The one it raises with no
+ * cause, for a capacity past what a vector can hold, has no fatal error to find, so the reader
+ * falls back on it anyway.
+ */
+ private def isVectorGrowthFailure(e: Throwable): Boolean =
+ e.getClass == classOf[RuntimeException] && e.getCause.isInstanceOf[OutOfMemoryError]
+
+ /**
+ * Throws a pending task kill, for the reader's loops that can run long without returning to
+ * `FileScanRDD`, which is where a plain read meets one.
+ */
+ def throwIfKilled(): Unit = throwIfKilled(null)
+
+ /**
+ * Throws a pending task kill, wrapped in a checked [[SparkException]] around `surfaced`, the
+ * error the kill surfaced as, if any. Under `ignoreCorruptFiles`, `FileScanRDD` would log any
+ * `RuntimeException` or `IOException` from a reader as a corrupt file and go on to open the
+ * split's remaining files, since it checks for a kill only between batches. The executor reports
+ * the task as killed either way. This covers the kills this reader sees at its own checks and
+ * catches. One that surfaces as an `IOException` from `readFilteredRowGroup` or from phase 2's
+ * decoding reaches that catch as it does for a plain read.
+ *
+ * The executor reports a task as killed only for an `InterruptedException` or a `NonFatal`
+ * error. So a `surfaced` error of any other type goes out as it is, the way a plain read raises
+ * it when decoding meets the same error. For example a codec's `InternalError`, or a native
+ * codec's `UnsatisfiedLinkError`. One from evaluating the filter goes out as it is too. Only a
+ * built-in expression can raise one there, such as an `ExceptionInInitializerError`, since
Review Comment:
Minor: the premise I gave in R10-8, which this sentence now carries, does not hold for a DSv2 function. A catalog `ScalarFunction` resolves to `StaticInvoke` or `ApplyFunctionExpression` (`V2ExpressionUtils.resolveScalarFunction`), neither of which carries the `INVOKE` pattern that `InjectRuntimeFilter.isSimpleExpression` excludes, and neither wraps its errors. So a join on `cat.ns.fn(f.a) = d.k` can give `might_contain(bf, xxhash64(fn(a)))` as a storage filter, and a `NoClassDefFoundError` from `fn` goes out through `case other`. The behavior is the one this describes; only the reason is off. Could this keep "One from evaluating the filter goes out as it is too, such as an `ExceptionInInitializerError`" and drop the "Only a built-in expression ... since ..." part?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1084 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A byte child sized up front is
+ * charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}. A
+ * fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // decoders of their last page.
+ closeDataPages();
+ releaseRowGroupReaders();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (reportedMessages.add(OFFSET_INDEX_UNREADABLE)) {
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn(OFFSET_INDEX_UNREADABLE, MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long survivors = 0L;
+ long firstSurvivorAt = 0L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_DECODING, true);
+ // The failed decode may have grown the scratch vectors, and phase 2 reads the same pages
+ // into vectors of its own. So capacity-sized ones replace them, and beside a plain read's
+ // own vectors the reader then holds one capacity-sized batch of key vectors.
+ closeAll(keyScratchVectors);
+ allocateKeyScratch();
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = predicate.eval(keyScratchBatch.getRow(r));
+ } catch (Throwable t) {
+ // Fall back on anything but a task kill or a fatal error. The predicate runs here without
+ // the conjuncts before it in the plan, so it can throw on a row a plain scan never
+ // evaluates it on (see `ParquetStorageFilter.isSupportedStorageFilter`). Giving the
+ // filter up returns every row the pushed filter kept, and the post-scan Filter decides
+ // them in its own order, raising the same error for a row it does evaluate.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK exception
+ // (`timestamp_seconds` of a decimal) or an `AssertionError`, and a Hive UDF throws a
+ // checked SparkException. Rethrowing would fail a query a plain read answers, or drop
+ // the rest of the file silently under `ignoreCorruptFiles`.
+ //
+ // A fatal error goes out even on a row an earlier conjunct of the plan would have
+ // rejected, so the reader never runs on after one the executor would take as fatal.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_EVALUATION, false);
+ return null;
+ }
+ if (!survives) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ long rowsBefore = pushedFilterRowCount - remaining + r;
+ if (survivors++ == 0) firstSurvivorAt = rowsBefore;
+ if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ if (currentKeyAccumulators == null) {
+ // The survivors from this one on, at the rate seen since the first one, so that rows
+ // before it, which a clustered key leaves, do not dilute the rate.
+ long expected = (long) Math.ceil(
+ (double) survivors * (remaining - r) / (rowsBefore - firstSurvivorAt + 1));
+ splicedBytes += allocateKeyAccumulators(cap - splicedBytes - rangeBytes, expected);
+ }
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // The buffer is this reader's own and optional, so failing to fill it gives splicing up
+ // rather than the filter, and the memory goes back at once. What fails here:
+ // - a survivor vector that cannot grow;
+ // - a key value the predicate did not read on this row, such as `b` in
+ // `coalesce(a, b)`, since the copy decodes every key;
+ // - a bug of ours, which only the WARN then shows.
+ // Phase 2 reads the key columns the way a plain read does, so a value that cannot be
+ // decoded fails the read only where a plain read fails.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together.
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, and returns the charge for the
+ * byte children it sized up front. The rest of the set is charged per row as survivors arrive.
+ * A variable-length key's byte child is reserved for what the set queued before it held, scaled
+ * to the {@code expectedRows} survivors this set can expect, where that is more than its default
+ * allocation. So it does not grow one doubling at a time for every capacity's worth of
+ * survivors, and a sparse last set is not sized as a full one. With its default doubling,
+ * {@code reserve} takes twice what it is asked for, so with its null bytes the child takes the
+ * factor's four bytes per value byte. That happens only when the charge fits within
+ * {@code headroom}, what the cap leaves, together with the fixed part and a range of each of
+ * those rows, since a scattered survivor adds a range. So the rows the set is sized for stay
+ * within the cap. The sizing is a prediction, though. A set sized for values longer than the
+ * ones that then arrive keeps the unused part charged, since its child holds that memory. So a
+ * larger cap can still give splicing up where a smaller one, which grew the set instead, keeps
+ * splicing. That costs one more read of the key columns, never a different answer. The memory
+ * is taken before any value arrives, so it is charged now, as if those values had arrived, and
+ * the values written into it use that charge up before they add to it.
+ */
+ private long allocateKeyAccumulators(long headroom, long expectedRows) {
Review Comment:
Minor, optional: in hindsight, the up-front sizing I asked for in R7 may not be worth what it now costs. It saves only a later set's regrowth copies, for a variable-length key with more than `capacity` survivors in a row group: per set at capacity 4096, about 460 KB copied and 400 KB allocated for 36-byte keys, or about 9 MB copied for 1000-byte keys, while phase 1 decodes and hashes every row's key. The benchmark, with long keys, never reaches it. For that it now takes about 56 lines here (`prepaidValueBytes`, the survivor bookkeeping, the headroom check and this javadoc), two tests of its own, and more memory than growing: 16,384,000 bytes for a full set of 1000-byte keys against 9,212,000 grown. Every cap window since R9-6 also came from its sized-or-not branch, including the value-length drift now documented. If every set grew the way each row group's first set already does, charged by the factor per value, the charge would never depend on the cap and never be above today's, s
o a larger cap could never splice less. Only the fixed-part test (L1228, 4000 against 1000) and the sparse-last-set test (L1279, 8000 against 6000) would need re-pinning, and the 32142 and scattered tests pass as they are. Would you consider dropping it? Keeping it as it is is fine too.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3742 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t: VirtualMachineError => fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the survivors it can expect and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the survivors it can still expect at the rate seen since the row group's
+ // first survivor. That memory is taken before any value arrives, so it is charged then. Six
+ // survivors over a capacity of 4 fill one set of four 1000-byte keys, and the second set starts
+ // two rows before the end of the row group, with every row since the first survivor kept. So
+ // it is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000
+ // bytes each, so they add 8000 more once the sizing is used up, and the row group holds 32142
+ // bytes in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or
+ // a double charge would each get wrong. The second set's byte child is reserved for 2000
+ // bytes, which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000.
+ // And the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ assertSpliceCharge(32142L, rows = 6) { cap =>
+ readUnderCap(path, laterSetSchema, laterSetFilter, cap, batchSize = 4)
+ }
+
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ if (cap >= 32142L) {
+ assert(secondSet == Seq(4000),
+ s"the second set must be sized for the two values left under cap $cap; got $secondSet")
+ }
+ }
+ }
+ }
+
+ test("a later accumulator set is sized only when it fits with its rows' fixed part and ranges") {
+ // The second set's two keys are 500 bytes here, shorter than the first set's 1000, so growing
+ // value by value is the cheaper path: 20142 bytes in all against 24142 sized. A cap of 24222
+ // sizes the set, whose byte child is then reserved for 2000 bytes and holds 4000. One of
+ // 24221 leaves 8113 bytes after the first set and its range, a byte short of the 8000 of
+ // sizing, the 34 of the two rows' fixed part and the 80 of a range for each. So the set grows
+ // instead, to a 1000-byte child, and keeps splicing. A cap of 20141 gives splicing up.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 500)
+ def run(maxSplicedBytes: String): Long =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)._2
+
+ val uncapped = run("64MB")
+ assert(run("24222b") == uncapped, "a cap that fits the sizing must keep splicing")
+ assert(run("24221b") == uncapped, "and so must one that only fits growing value by value")
+ assert(run("20141b") > uncapped, "a cap under both must give splicing up")
+
+ Seq(Long.MaxValue -> 4000, 24222L -> 4000, 24221L -> 1000).foreach { case (cap, child) =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected, s"every survivor must come back with its own key under cap $cap")
+ assert(secondSet == Seq(child),
+ s"under cap $cap the second set's byte child must hold $child bytes; got $secondSet")
+ }
+ }
+ }
+
+ test("a later set is sized only when the ranges of its scattered survivors fit too") {
+ // Keys 9 to 12 fill the first set over a capacity of 4, and keys 14 and 17 come scattered, each
+ // in a range of its own. The second set opens at key 14, sized for the four rows the rate
+ // since the first survivor promises, and its sizing is charged up front. So it is sized only
+ // when the cap holds a range for each of those rows as well. Without that, caps of 32216 to
+ // 32221 sized the set and then passed the cap on key 17's range, giving splicing up where a
+ // smaller cap of 32215, which grows the set instead, kept it. For this file, every cap from
+ // the smallest that splices up must splice, and every cap returns the same six rows.
+ withTempDir { dir =>
+ val (path, _) = writeLaterSetFile(dir, lastTwoBytes = 1000)
+ val prefix = Substring(BoundReference(0, StringType, nullable = true), Literal(1), Literal(4))
+ val keys = Seq("0009", "0010", "0011", "0012", "0014", "0017").map(Literal(_))
+ def readUnder(cap: Long): (Int, Long) =
+ readUnderCap(path, laterSetSchema, Seq(In(prefix, keys)), s"${cap}b", batchSize = 4)
+ val (_, uncapped) = readUnder(64L * 1024 * 1024)
+ val (smallerRows, smaller) = readUnder(24221L)
+ assert(smallerRows == 6 && smaller > uncapped, "the smallest cap that splices must be 24222")
+ Seq(24222L, 32215L, 32216L, 32221L, 32222L).foreach { cap =>
+ val (rows, bytes) = readUnder(cap)
+ assert(rows == 6 && bytes == uncapped,
+ s"a cap of $cap must keep the six rows and splicing; got $rows rows")
+ }
+ }
+ }
+
+ test("a sparse last set is sized for the survivors it can expect") {
+ // Every tenth of 125 keys survives, so over a capacity of 4 the fourth set holds key 120 alone.
+ // It opens five rows before the end of the row group, at a rate of 13 survivors in 121 rows, so
+ // it expects one survivor and is sized for one 1000-byte value, which `reserve` doubles to a
+ // 2000-byte child. Sized for the five rows left, as a full set, it would have taken 8000. The
+ // two sets before it are sized as full ones, since the rate promises them four survivors each.
+ withTempDir { dir =>
+ val df = spark.range(0, 125).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), REPEAT('x', 996)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 1024L * 1024)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val lastDigit =
+ Substring(BoundReference(0, StringType, nullable = true), Literal(4), Literal(1))
+ val (rows, reader) = readAllWith(path, Seq("k", "v"),
+ createFilter(Seq(EqualTo(lastDigit, Literal("0"))), laterSetSchema),
+ (b, i) => (b.column(0).getUTF8String(i).toString.take(4),
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ assert(rows.map(_._1) == (0 to 120 by 10).map(id => f"$id%04d"), s"got ${rows.map(_._1)}")
+ val children = rows.map(_._2)
+ assert(children.slice(4, 12).distinct == Seq(8000) && children.drop(12) == Seq(2000),
Review Comment:
Minor: this cannot tell "sized for its one survivor" from "not sized at all", since both end with a 2000-byte child: `reserve(1000)` doubles to 2000, and growing from the 16-byte default reaches 2000 too. So it passes if the last set's sizing were skipped, or if `Math.ceil` at `LateMaterializationParquetRecordReader.java:812` became `floor` (65/121 would give 0), and no other test pins the rounding. With `spark.range(0, 135)`, the last set holds keys 120 and 130 and expects ceil(195/121) = 2, so it reserves 2000 and holds 4000, which not sizing (2000), `floor` (2000) and a full set (8000) all miss, while sets 2 and 3 stay at 8000. Could it use that, with `children.drop(12).distinct == Seq(4000)`?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3742 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t: VirtualMachineError => fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the survivors it can expect and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the survivors it can still expect at the rate seen since the row group's
+ // first survivor. That memory is taken before any value arrives, so it is charged then. Six
+ // survivors over a capacity of 4 fill one set of four 1000-byte keys, and the second set starts
+ // two rows before the end of the row group, with every row since the first survivor kept. So
+ // it is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000
+ // bytes each, so they add 8000 more once the sizing is used up, and the row group holds 32142
+ // bytes in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or
+ // a double charge would each get wrong. The second set's byte child is reserved for 2000
+ // bytes, which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000.
+ // And the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ assertSpliceCharge(32142L, rows = 6) { cap =>
+ readUnderCap(path, laterSetSchema, laterSetFilter, cap, batchSize = 4)
+ }
+
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ if (cap >= 32142L) {
+ assert(secondSet == Seq(4000),
+ s"the second set must be sized for the two values left under cap $cap; got $secondSet")
+ }
+ }
+ }
+ }
+
+ test("a later accumulator set is sized only when it fits with its rows' fixed part and ranges") {
+ // The second set's two keys are 500 bytes here, shorter than the first set's 1000, so growing
+ // value by value is the cheaper path: 20142 bytes in all against 24142 sized. A cap of 24222
+ // sizes the set, whose byte child is then reserved for 2000 bytes and holds 4000. One of
+ // 24221 leaves 8113 bytes after the first set and its range, a byte short of the 8000 of
+ // sizing, the 34 of the two rows' fixed part and the 80 of a range for each. So the set grows
+ // instead, to a 1000-byte child, and keeps splicing. A cap of 20141 gives splicing up.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 500)
+ def run(maxSplicedBytes: String): Long =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)._2
+
+ val uncapped = run("64MB")
+ assert(run("24222b") == uncapped, "a cap that fits the sizing must keep splicing")
+ assert(run("24221b") == uncapped, "and so must one that only fits growing value by value")
+ assert(run("20141b") > uncapped, "a cap under both must give splicing up")
+
+ Seq(Long.MaxValue -> 4000, 24222L -> 4000, 24221L -> 1000).foreach { case (cap, child) =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected, s"every survivor must come back with its own key under cap $cap")
+ assert(secondSet == Seq(child),
+ s"under cap $cap the second set's byte child must hold $child bytes; got $secondSet")
+ }
+ }
+ }
+
+ test("a later set is sized only when the ranges of its scattered survivors fit too") {
+ // Keys 9 to 12 fill the first set over a capacity of 4, and keys 14 and 17 come scattered, each
+ // in a range of its own. The second set opens at key 14, sized for the four rows the rate
+ // since the first survivor promises, and its sizing is charged up front. So it is sized only
+ // when the cap holds a range for each of those rows as well. Without that, caps of 32216 to
+ // 32221 sized the set and then passed the cap on key 17's range, giving splicing up where a
+ // smaller cap of 32215, which grows the set instead, kept it. For this file, every cap from
+ // the smallest that splices up must splice, and every cap returns the same six rows.
+ withTempDir { dir =>
+ val (path, _) = writeLaterSetFile(dir, lastTwoBytes = 1000)
+ val prefix = Substring(BoundReference(0, StringType, nullable = true), Literal(1), Literal(4))
+ val keys = Seq("0009", "0010", "0011", "0012", "0014", "0017").map(Literal(_))
+ def readUnder(cap: Long): (Int, Long) =
+ readUnderCap(path, laterSetSchema, Seq(In(prefix, keys)), s"${cap}b", batchSize = 4)
+ val (_, uncapped) = readUnder(64L * 1024 * 1024)
+ val (smallerRows, smaller) = readUnder(24221L)
+ assert(smallerRows == 6 && smaller > uncapped, "the smallest cap that splices must be 24222")
+ Seq(24222L, 32215L, 32216L, 32221L, 32222L).foreach { cap =>
+ val (rows, bytes) = readUnder(cap)
+ assert(rows == 6 && bytes == uncapped,
+ s"a cap of $cap must keep the six rows and splicing; got $rows rows")
+ }
+ }
+ }
+
+ test("a sparse last set is sized for the survivors it can expect") {
+ // Every tenth of 125 keys survives, so over a capacity of 4 the fourth set holds key 120 alone.
+ // It opens five rows before the end of the row group, at a rate of 13 survivors in 121 rows, so
+ // it expects one survivor and is sized for one 1000-byte value, which `reserve` doubles to a
+ // 2000-byte child. Sized for the five rows left, as a full set, it would have taken 8000. The
+ // two sets before it are sized as full ones, since the rate promises them four survivors each.
+ withTempDir { dir =>
+ val df = spark.range(0, 125).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), REPEAT('x', 996)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 1024L * 1024)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val lastDigit =
+ Substring(BoundReference(0, StringType, nullable = true), Literal(4), Literal(1))
+ val (rows, reader) = readAllWith(path, Seq("k", "v"),
+ createFilter(Seq(EqualTo(lastDigit, Literal("0"))), laterSetSchema),
+ (b, i) => (b.column(0).getUTF8String(i).toString.take(4),
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ assert(rows.map(_._1) == (0 to 120 by 10).map(id => f"$id%04d"), s"got ${rows.map(_._1)}")
+ val children = rows.map(_._2)
+ assert(children.slice(4, 12).distinct == Seq(8000) && children.drop(12) == Seq(2000),
+ s"the sparse last set must be sized for its one survivor; got $children")
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ assertSpliceCharge(166L, rows = 7) { cap =>
+ readUnderCap(path, schema, storageFilters, cap, batchSize = 16)
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warnings) = withLogged(partialReadWarning) {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(warnings == 0,
+ s"the footer check must get there before the exception path; got $warnings")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warnings) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(warnings == 0, "the footer check must decline the row group rather than the read " +
+ s"discovering it; got $warnings")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize = 64L * 1024 * 1024,
+ options = Map(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT -> "100"))
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ // The rows a reader hands back before it throws, what it throws, and, for the filtering
+ // reader, whether its scratch vectors were replaced on the way and the old ones freed. Both
+ // read off heap, where an old scratch vector left open would leak native memory.
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable, Boolean) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(true, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(true, 16, storageFilter)
+ }
+ def scratch: Seq[WritableColumnVector] = reader match {
+ case filtering: LateMaterializationParquetRecordReader =>
+ ParquetTestAccess.keyScratchVectors(filtering)
+ case _ => Nil
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val scratchBefore = scratch
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ val replacedAndFreed = scratch.zip(scratchBefore).exists { case (now, before) =>
+ (now ne before) && before.asInstanceOf[OffHeapColumnVector].valuesNativeAddress() == 0
Review Comment:
Nit: this cast chain appears again at L1662 and L1687, while the off-heap close test has `address` (L3077-3078), and the hook at L1658 and L1666 looks up `keyScratchVectors` in both branches. Also, `valuesNativeAddress()` returns `data`, which `OffHeapColumnVector.reserveInternal` leaves at 0 for a string or binary vector, so a string-key variant of these checks would pass `== 0` on a leak and fail `!= 0`. They all use long keys today. Could one suite-level `address`, noted as fixed-width only, serve all four, with one lookup before the hook's `if`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4186439490
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A byte child sized up front is
+ * charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}. A
+ * fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // dictionary and decoders of their last page.
+ closeDataPages();
+ releaseRowGroupReaders();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (reportedMessages.add(OFFSET_INDEX_UNREADABLE)) {
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn(OFFSET_INDEX_UNREADABLE, MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_DECODING, true);
+ // The failed decode may have grown the scratch vectors, and phase 2 reads the same pages
+ // into vectors of its own. So capacity-sized ones replace them, and beside a plain read's
+ // own vectors the reader then holds one capacity-sized batch of key vectors.
+ closeAll(keyScratchVectors);
+ allocateKeyScratch();
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = predicate.eval(keyScratchBatch.getRow(r));
+ } catch (Throwable t) {
+ // Fall back on anything but a task kill or a fatal error. The predicate runs here without
+ // the conjuncts before it in the plan, so it can throw on a row a plain scan never
+ // evaluates it on (see `ParquetStorageFilter.isSupportedStorageFilter`). Giving the
+ // filter up returns every row the pushed filter kept, and the post-scan Filter decides
+ // them in its own order, raising the same error for a row it does evaluate.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK exception
+ // (`timestamp_seconds` of a decimal) or an `AssertionError`, and a Hive UDF throws a
+ // checked SparkException. Rethrowing would fail a query a plain read answers, or drop
+ // the rest of the file silently under `ignoreCorruptFiles`.
+ //
+ // A fatal error goes out even on a row an earlier conjunct of the plan would have
+ // rejected, so the reader never runs on after one the executor would take as fatal.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_EVALUATION, false);
+ return null;
+ }
+ if (!survives) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ if (currentKeyAccumulators == null) {
+ splicedBytes +=
+ allocateKeyAccumulators(cap - splicedBytes - rangeBytes, remaining - r);
+ }
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // The buffer is this reader's own and optional, so failing to fill it gives splicing up
+ // rather than the filter, and the memory goes back at once. What fails here:
+ // - a survivor vector that cannot grow;
+ // - a key value the predicate did not read on this row, such as `b` in
+ // `coalesce(a, b)`, since the copy decodes every key;
+ // - a bug of ours, which only the WARN then shows.
+ // Phase 2 reads the key columns the way a plain read does, so a value that cannot be
+ // decoded fails the read only where a plain read fails.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together.
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, and returns the charge for the
+ * byte children it sized up front. The rest of the set is charged per row as survivors arrive.
+ * A variable-length key's byte child is reserved for what the set queued before it held, scaled
+ * to the {@code rowsLeft} rows that can still reach this set, where that is more than its default
+ * allocation. So it does not grow one doubling at a time for every capacity's worth of
+ * survivors. With its default doubling, {@code reserve} takes twice what it is asked for, so
+ * with its null bytes the child takes the factor's four bytes per value byte. That happens only
+ * when the charge and the fixed part of those rows fit within {@code headroom}, what the cap
+ * leaves. The memory is taken before any value arrives, so it is charged now, as if those values
+ * had arrived, and the values written into it use that charge up before they add to it.
+ */
+ private long allocateKeyAccumulators(long headroom, long rowsLeft) {
+ WritableColumnVector[] previous = survivorBatches.peekLast();
+ int rows = (int) Math.min(capacity, rowsLeft);
Review Comment:
**The `remaining - r` bound I suggested in R9-6 is imprecise: it counts the rows left to evaluate, not the survivors still to come.** So it helps only when the survivors fill the row group's tail, which is the shape of the re-pinned tests (`k >= "0015"` at `ParquetStorageFilterSuite.scala:1159`, and `k >= 394`). At the selectivities this is for, a last set still opens with at least `capacity` rows left, so `rows` is `capacity` and the set is reserved and charged as a full one. At capacity 4096 with 1000-byte keys and 1% survivors in a 1M-row row group, the last set opens about 180k rows before the end and takes 16,384,000 bytes for 1,808 values, where sizing by survivors would take 7,232,000.
The headroom I suggested also leaves out the 40 range bytes (L801) that each later scattered survivor adds, so a larger cap can still size a set and then give splicing up. Keeping rows 8-11, 13 and 16 of `writeLaterSetFile(dir, 1000)` at capacity 4, the second set opens at row 13 with 7 rows left and is reserved 4000 value bytes for two values. Caps 24222 to 32215 splice, 32216 to 32221 give splicing up at row 16 and read the key column twice, and 32222 splices again.
Could the check at L908 also reserve `ROW_RANGE_BYTES` per row, which closes that window within a set, and could `rows` scale with the survival rate this row group has shown so far (`survivorBatches.size() * capacity` survivors over the rows evaluated), which removes the over-reservation? A scattered fixture like the one above would pin both.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/StorageFilterPushdownBenchmark.scala:
##########
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import java.io.File
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetOutputFormat}
+import org.apache.parquet.hadoop.util.HadoopInputFile
+
+import org.apache.spark.SparkConf
+import org.apache.spark.benchmark.Benchmark
+import org.apache.spark.sql.{DataFrame, SparkSession}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
+import org.apache.spark.sql.catalyst.expressions.{BloomFilterMightContain, Literal, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.classic.ExpressionUtils.column
+import org.apache.spark.sql.execution.{FileSourceScanExec, SparkPlan}
+import org.apache.spark.sql.execution.datasources.parquet.StorageFilterMetrics
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.BinaryType
+import org.apache.spark.util.Utils
+
+/**
+ * Benchmark to measure storage-filter pushdown in the vectorized Parquet reader. Every case runs
+ * with `spark.sql.parquet.storageFilterPushdown.enabled` off and on, in two forms:
+ * - the join whose runtime bloom filter is the storage filter, which is the path the feature
+ * takes. The join's shuffle and sort are the same in both arms, and they grow with the rows
+ * that survive, so they dilute what the reader saves or costs;
+ * - a scan of the fact table alone under the same bloom filter, built once outside the timed
+ * body, which shows the reader's own share.
+ *
+ * Every case reads a fact table of about 100 MB in 34 row groups. The cases differ in four things:
+ * - the width of a row. One table has 50,000 rows of a long key and a 2 kB value that does not
+ * compress. The other has 1.4M rows of a long key and eight hashed long values;
+ * - the page size. Each table is written twice, in 8 kB pages and in parquet's default 1 MB
Review Comment:
Nit: the narrow table's default pages are not 1 MB. Its 42,000-row row groups are cut by parquet's default page row limit of 20,000 (`ParquetProperties.DEFAULT_PAGE_ROW_COUNT_LIMIT`), so its pages hold 20,000, 20,000 and 2,000 rows, about 160 kB per long column, and the section title's "about 14000 rows" is an average that no page has. Could this say "parquet's default pages, 1 MB or 20,000 rows, whichever comes first"?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import scala.util.control.NonFatal
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while a code-generated `created`
+ // holds the deserialized filter instead. Dropping the expression lets a columnar scan's task
+ // reclaim those bytes for the rest of its life. An interpreted `created`, the fallback when
+ // codegen fails or is off, keeps the expression itself, so the bytes stay. A row-based scan
+ // keeps them anyway, since its closure captures the scan node, whose storage filters hold the
+ // same bytes.
+ // Safe because every task attempt deserializes its own copy from the driver's bytes, and this
+ // lazy val is the only reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while applying a storage filter, if it must not fall back on
+ * it by giving the filter up. `fromRead` says the error came from decoding the key pages rather
+ * than from evaluating the filter. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped when the error is one the executor's kill case takes,
+ * see [[throwIfKilled]]. The error may be the kill's interrupt surfacing through a UDF that
+ * wrapped it. `PythonRunner` reads a kill off the task context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had a plain plan raised `e`, goes out as itself, so the executor finds
+ * it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in an `UnknownError`, a fatal error of another type.
+ * `FileScanRDD` would take a bare `InternalError` from a reader for a corrupt file under
+ * `ignoreCorruptFiles`. A `NonFatal` wrapper does not work either. With a kill pending, the
+ * executor reports one as killed and keeps running. A plain plan raises it from the post-scan
+ * Filter, where neither happens, and the executor stops. It stops on the `UnknownError` too, at
+ * any `spark.executor.killOnFatalError.depth` of 1 or more, since that is at the top.
+ *
+ * An error from decoding differs in three ways:
+ * - a plain read raises it from the reader, where `FileScanRDD` wraps a `NonFatal` one in
+ * FAILED_READ_FILE before the executor looks, so a fatal error deeper in the chain has to sit
+ * one level shallower to count. Under `ignoreCorruptFiles` a plain read may skip the rest of
+ * the file on it instead.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ *
+ * Anything else falls back. An evaluation error is the filter's own, which the post-scan Filter
+ * raises again for a row it does evaluate. A decoding error is met again by the plain read of
+ * the same pages.
+ */
+ def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new UnknownError("A storage filter met a fatal error it must not absorb")
+ .initCause(internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ if (Executor.isFatalError(e, if (fromRead) fatalErrorDepth - 1 else fatalErrorDepth)) {
Review Comment:
Minor: this lowers the depth for every decoding error, but `attachFilePath` wraps only a `NonFatal` one in FAILED_READ_FILE (`FileDataSourceV2.scala:141-143`), as the scaladoc at L267-268 says. A `LinkageError` goes through unwrapped, so for one whose chain holds a fatal error this searches one level short of where the executor looks. With `spark.executor.killOnFatalError.depth=3` and a phase-1 decode that raises `ExceptionInInitializerError(RuntimeException(OutOfMemoryError))`, this checks `isFatalError(e, 2)`, misses the OOM and falls back. Phase 2 then meets a `NoClassDefFoundError` without the OOM, so the task fails and the executor keeps running, whereas a plain read hands the EIIE to the executor, which finds the OOM at level 3 and stops. The trigger is narrow, and at the default depth both find it. Could this be `if (fromRead && NonFatal(e)) fatalErrorDepth - 1 else fatalErrorDepth`? Every assertion of the policy test still holds with it.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -476,34 +483,99 @@ private boolean containsPath(Type parquetType, String[] path, int depth) {
return false;
}
- private void checkEndOfRowGroup() throws IOException {
- if (rowsReturned != totalCountLoadedSoFar) return;
+ /**
+ * Loads the row group the read continues with, and returns false if there is none left. Only a
+ * subclass that picks its own rows can answer false, because this class reads every row group
+ * parquet hands it, and then {@code totalRowCount} already says when they run out.
+ */
+ protected boolean loadNextRowGroup() throws IOException {
PageReadStore pages = reader.readNextRowGroup();
if (pages == null) {
throw new IOException("expecting more rows but reached last block. Read "
+ rowsReturned + " out of " + totalRowCount);
}
+ // Null ranges, because this path never picks the rows itself, and whether parquet narrowed the
+ // row group by a pushed filter's column index is known only to the store it handed back.
+ installRowGroup(pages, null);
+ return true;
+ }
+
+ /**
+ * Points the column readers at one row group's {@code pages}, from which they are to read the
+ * rows {@code rowRanges} names. Null ranges leave that to the store, which names its rows through
+ * its row indexes when parquet filtered it and otherwise holds every row of the row group.
+ */
+ protected final void installRowGroup(PageReadStore pages, RowRanges rowRanges)
+ throws IOException {
if (rowIndexGenerator != null) {
rowIndexGenerator.initFromPageReadStore(pages);
}
- for (ParquetColumnVector cv : columnVectors) {
- initColumnReader(pages, cv);
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (suppliesOwnVector(i)) {
+ // Cleared rather than left alone, because a reader a previous row group set would otherwise
+ // be driven over this row group's pages.
+ columnVectors[i].setColumnReader(null);
+ } else {
+ initColumnReader(pages, rowRanges, columnVectors[i]);
+ }
}
totalCountLoadedSoFar += pages.getRowCount();
}
- private void initColumnReader(PageReadStore pages, ParquetColumnVector cv) throws IOException {
+ /**
+ * Drops the column readers of the row group installed last, and the row-index iterator, both of
+ * which hold the row ranges that row group was read over. For a subclass that does work of its
+ * own between two row groups, so that work does not run with the previous ranges still held.
+ * Call it only from {@link #loadNextRowGroup}, once the row group is fully emitted. A batch read
+ * after it needs another row group installed first.
+ */
+ protected final void releaseRowGroupReaders() {
Review Comment:
Nit, on the PR description: it still says this class "is 595 lines against master's 543" and "ends up with six protected methods, four of them new, and calls every one of them itself". With this method it is 615 lines with seven protected methods, five of them new, and only the subclass calls this one (`LateMaterializationParquetRecordReader.java:499`). Also, "the scan alone is up to 20% slower" reads the 0.8X column as time, but it is throughput: the JDK 17 case takes 140 ms against 114 ms, 1.23x the time, and the JDK 21 one 1.28x. Could the description be updated, since it becomes the commit message?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
+ try {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ rethrow(under(2, oom), fromRead = false)
Review Comment:
Nit: if `rethrowIfMustPropagate` regressed, this call and L677, as well as the renamed L661, L665, L678 and L696, would throw an `OutOfMemoryError`, `InternalError` or `UnknownError`. ScalaTest aborts the whole suite on a `VirtualMachineError` instead of failing this test, which L1620-1622 already guard against. Could a small `fallsBack(e, fromRead)` catch `VirtualMachineError` and call `fail`, as there?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A byte child sized up front is
+ * charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}. A
+ * fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // dictionary and decoders of their last page.
Review Comment:
Nit: a lazily decoded column's batch vector keeps the dictionary too (`column.setDictionary(new ParquetDictionary(...))` at `VectorizedColumnReader.java:308`), and `WritableColumnVector.reset()` does not clear it, so releasing the readers does not free it until a later `readBatch` replaces it. A dictionary also belongs to the column chunk rather than its last page. Could this end with "and the decoders of their last page"?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
+ try {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ rethrow(under(2, oom), fromRead = false)
+ } finally {
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, depth.toString)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ rethrow(new InternalError("codec"), fromRead = true)
+ rethrow(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ rethrow(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ rethrow(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val withDefault = new MetadataBuilder()
+ .putString(ResolveDefaultColumns.EXISTS_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .putString(ResolveDefaultColumns.CURRENT_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .build()
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = true, withDefault), StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split") {
+ // Two row groups give the filter up on an evaluation error, and the reader reports that once.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val rowGroups = footerOf(path).getBlocks.asScala.map(_.getRowCount)
+ assert(rowGroups.size == 2 && rowGroups.head >= 5L && rowGroups.head < 150L,
+ s"keys 5 and 150 must fall in two row groups; got $rowGroups")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val filter = createFilter(Seq(And(failingAt(5L), failingAt(150L))), kvSchema())
+ val ((result, reader), warnings) =
+ withLogged("applying it to a row raised an error")(readAll(path, filter))
+ try {
+ assert(result == rows, s"both row groups must come back whole; got ${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ val (_, sevenUncapped, _) = run("64MB", 394L)
+ val (_, sevenAtCharge, _) = run("103b", 394L)
+ val (_, sevenUnderCharge, _) = run("102b", 394L)
+ assert(sevenAtCharge == sevenUncapped,
+ s"a cap of exactly the charge must keep splicing; got $sevenAtCharge against " +
+ sevenUncapped)
+ assert(sevenUnderCharge > sevenUncapped,
+ s"a cap one byte under it must give splicing up; got $sevenUnderCharge against " +
+ sevenUncapped)
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the rows left and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the rows that can still reach it. That memory is taken before any value
+ // arrives, so it is charged then. Six survivors over a capacity of 4 fill one set of four
+ // 1000-byte keys, and the second set starts two rows before the end of the row group. So it
+ // is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000 bytes
+ // each, so they add 8000 more once the sizing is used up, and the row group holds 32142 bytes
+ // in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or a
+ // double charge would each get wrong. The second set's byte child is reserved for 2000 bytes,
+ // which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000. And
+ // the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)
+
+ val (uncappedRows, uncappedBytes) = run("64MB")
+ val (atChargeRows, atChargeBytes) = run("32142b")
+ assert(uncappedRows == 6 && atChargeRows == 6,
+ s"the filter keeps keys 15..20 either way; got $uncappedRows and $atChargeRows")
+ assert(atChargeBytes == uncappedBytes,
+ "a cap of exactly the row group's charge must keep splicing and read the key column " +
+ s"once; got $atChargeBytes against $uncappedBytes")
+ val (_, underChargeBytes) = run("32141b")
+ assert(underChargeBytes > uncappedBytes,
+ "a cap one byte under it must give splicing up and read the key column again; " +
+ s"got $underChargeBytes against $uncappedBytes")
+
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ if (cap >= 32142L) {
+ assert(secondSet == Seq(4000),
+ s"the second set must be sized for the two values left under cap $cap; got $secondSet")
+ }
+ }
+ }
+ }
+
+ test("a later accumulator set is sized only when it fits with its rows' fixed part") {
+ // The second set's two keys are 500 bytes here, shorter than the first set's 1000, so growing
+ // value by value is the cheaper path: 20142 bytes in all against 24142 sized. A cap of 24142
+ // sizes the set, whose byte child is then reserved for 2000 bytes and holds 4000. One of
+ // 24141 leaves 8033 bytes after the first set and its range, a byte short of the 8000 of
+ // sizing and the 34 of the two rows' fixed part. So the set grows instead, to a 1000-byte
+ // child, and keeps splicing. A cap of 20141 gives splicing up.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 500)
+ def run(maxSplicedBytes: String): Long =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)._2
+
+ val uncapped = run("64MB")
+ assert(run("24142b") == uncapped, "a cap that fits the sizing must keep splicing")
+ assert(run("24141b") == uncapped, "and so must one that only fits growing value by value")
+ assert(run("20141b") > uncapped, "a cap under both must give splicing up")
+
+ Seq(Long.MaxValue -> 4000, 24142L -> 4000, 24141L -> 1000).foreach { case (cap, child) =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected, s"every survivor must come back with its own key under cap $cap")
+ assert(secondSet == Seq(child),
+ s"under cap $cap the second set's byte child must hold $child bytes; got $secondSet")
+ }
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, schema, storageFilters, maxSplicedBytes, batchSize = 16)
+ val (_, uncapped) = run("64MB")
+ val (rows, atCharge) = run("166b")
+ val (_, underCharge) = run("165b")
+ assert(rows == 7, s"the filter keeps keys 394..400; got $rows")
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Two row groups of 100 rows with `v = k * 10`, and a pushed `v IN (x, 1500)`, where `x` falls
+ // between the first two pages of the first row group. Its statistics keep that row group, while
+ // its column index keeps no page of it.
+ private def writeColumnIndexEmptiedFile(dir: File): (String, Seq[sources.Filter]) = {
+ val blocks =
+ (0 until 2).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ (path, Seq(sources.In("v", Array[Any](valueBetweenFirstTwoPages(path), 1500L))))
+ }
+
+ // A value of the long column `v` that falls between the first two pages of the first row group.
+ private def valueBetweenFirstTwoPages(filePath: String): Long =
+ Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val index = reader.readColumnIndex(reader.getRowGroups.get(0).getColumns.get(1))
+ def asLong(bytes: java.nio.ByteBuffer): Long =
+ bytes.duplicate().order(java.nio.ByteOrder.LITTLE_ENDIAN).getLong(0)
+ val firstMax = asLong(index.getMaxValues.get(0))
+ val secondMin = asLong(index.getMinValues.get(1))
+ assert(secondMin - firstMax > 1,
+ s"no gap between the first two pages: $firstMax, $secondMin")
+ firstMax + 1
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize = 64L * 1024 * 1024,
+ options = Map(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT -> "100"))
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ // The rows a reader hands back before it throws, what it throws, and, for the filtering
+ // reader, whether its scratch vectors were replaced on the way.
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable, Boolean) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(false, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(false, 16, storageFilter)
+ }
+ def scratch: Seq[WritableColumnVector] = reader match {
+ case filtering: LateMaterializationParquetRecordReader =>
+ ParquetTestAccess.keyScratchVectors(filtering)
+ case _ => Nil
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val scratchBefore = scratch
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ (read.toSeq, error, scratch.zip(scratchBefore).exists { case (a, b) => a ne b })
Review Comment:
Minor: this checks only that the scratch vectors were replaced, by identity, on heap, with a long key in a one-row-group file. So dropping `closeAll(keyScratchVectors)` at `LateMaterializationParquetRecordReader.java:766` still passes here, while off heap every decoding fallback would leak a batch of native key vectors for the executor's life. The other decoding-fallback test (L1597) also reads on heap with one row group, so the next row group's phase 1 over the rebuilt `keyScratchBatch` is not exercised either. Could `readUntilFailure` build the filtering reader off heap and assert that the old vectors' native address is 0, over a file with a row group after the damaged one?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTestAccess.scala:
##########
@@ -173,17 +187,74 @@ object ParquetTestAccess {
try { deltaLengthByteArrayCtor.newInstance() }
catch { case e: ReflectiveOperationException => throw rethrow(e) }
+ // -------- LateMaterializationParquetRecordReader --------
+
+ private val keyScratchVectorsField = {
Review Comment:
Nit: these lookups of private reader state are eager vals, and this object is also loaded by `VectorizedRleValuesReaderSuite`, `ParquetVectorUpdaterSuite` and four benchmarks. So renaming one of these members, most likely this round's own `keyScratchVectors`, fails those at object initialization as well. Could the new ones be lazy, or live in `ParquetStorageFilterSuite`, which does not run under `spark-submit --jars`?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import scala.util.control.NonFatal
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while a code-generated `created`
+ // holds the deserialized filter instead. Dropping the expression lets a columnar scan's task
+ // reclaim those bytes for the rest of its life. An interpreted `created`, the fallback when
+ // codegen fails or is off, keeps the expression itself, so the bytes stay. A row-based scan
+ // keeps them anyway, since its closure captures the scan node, whose storage filters hold the
+ // same bytes.
+ // Safe because every task attempt deserializes its own copy from the driver's bytes, and this
+ // lazy val is the only reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while applying a storage filter, if it must not fall back on
+ * it by giving the filter up. `fromRead` says the error came from decoding the key pages rather
+ * than from evaluating the filter. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped when the error is one the executor's kill case takes,
+ * see [[throwIfKilled]]. The error may be the kill's interrupt surfacing through a UDF that
+ * wrapped it. `PythonRunner` reads a kill off the task context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had a plain plan raised `e`, goes out as itself, so the executor finds
+ * it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in an `UnknownError`, a fatal error of another type.
+ * `FileScanRDD` would take a bare `InternalError` from a reader for a corrupt file under
+ * `ignoreCorruptFiles`. A `NonFatal` wrapper does not work either. With a kill pending, the
+ * executor reports one as killed and keeps running. A plain plan raises it from the post-scan
+ * Filter, where neither happens, and the executor stops. It stops on the `UnknownError` too, at
+ * any `spark.executor.killOnFatalError.depth` of 1 or more, since that is at the top.
+ *
+ * An error from decoding differs in three ways:
+ * - a plain read raises it from the reader, where `FileScanRDD` wraps a `NonFatal` one in
+ * FAILED_READ_FILE before the executor looks, so a fatal error deeper in the chain has to sit
+ * one level shallower to count. Under `ignoreCorruptFiles` a plain read may skip the rest of
+ * the file on it instead.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ *
+ * Anything else falls back. An evaluation error is the filter's own, which the post-scan Filter
+ * raises again for a row it does evaluate. A decoding error is met again by the plain read of
+ * the same pages.
+ */
+ def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new UnknownError("A storage filter met a fatal error it must not absorb")
+ .initCause(internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ if (Executor.isFatalError(e, if (fromRead) fatalErrorDepth - 1 else fatalErrorDepth)) {
+ // The one `Executor.isFatalError` found, which no SparkOutOfMemoryError can precede.
+ val fatal = Iterator.iterate(e)(_.getCause).find(Utils.isFatalError).get
+ if (!corrupt(fatal)) throw out(fatal)
+ }
+ }
+
+ /**
+ * Whether `e` is how `WritableColumnVector.reserve` reports a vector that ran out of memory to
+ * grow, a plain `RuntimeException` caused by an `OutOfMemoryError`. The one it raises with no
+ * cause, for a capacity past what a vector can hold, has no fatal error to find, so the reader
+ * falls back on it anyway.
+ */
+ private def isVectorGrowthFailure(e: Throwable): Boolean =
+ e.getClass == classOf[RuntimeException] && e.getCause.isInstanceOf[OutOfMemoryError]
+
+ /**
+ * Throws a pending task kill, for the reader's loops that can run long without returning to
+ * `FileScanRDD`, which is where a plain read meets one.
+ */
+ def throwIfKilled(): Unit = throwIfKilled(null)
+
+ /**
+ * Throws a pending task kill, wrapped in a checked [[SparkException]] around `surfaced`, the
+ * error the kill surfaced as, if any. Under `ignoreCorruptFiles`, `FileScanRDD` would log any
+ * `RuntimeException` or `IOException` from a reader as a corrupt file and go on to open the
+ * split's remaining files, since it checks for a kill only between batches. The executor reports
+ * the task as killed either way. This covers the kills this reader sees at its own checks and
+ * catches. One that surfaces as an `IOException` from `readFilteredRowGroup` or from phase 2's
+ * decoding reaches that catch as it does for a plain read.
+ *
+ * The executor reports a task as killed only for an `InterruptedException` or a `NonFatal`
+ * error. So a `surfaced` error of any other type goes out as it is, the way a plain read raises
+ * it where it meets the same error. For example a codec's `InternalError` while decoding, or a
+ * `LinkageError` from a UDF.
Review Comment:
Nit: "the way a plain read raises it" holds for a decoding error, not for an evaluation one. With a kill pending, a plain plan reports the task killed unless its post-scan Filter evaluates that row, while this reports it failed. And a storage filter cannot raise a UDF's `LinkageError`: its keys pass `InjectRuntimeFilter.isSimpleExpression`, and a Hive UDF wraps its errors in a `SparkException`. Could the sentence say decoding, with an example such as a native codec's `UnsatisfiedLinkError`?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
Review Comment:
Nit: this text is a literal here and again at L784, while the other counted texts are named (`partialReadWarning`, `splicingGivenUpOnError`). If `ROW_GROUP_GIVEN_UP_ON_EVALUATION` were reworded, L787 would fail and L784 would get the new text, but this copy could stay behind, and then L615 would pass even if the kill were logged as a give-up before it went out. Could both use one `private val` next to L246?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
Review Comment:
Nit: `withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") { ... }` (`SparkTestSuite.scala`) does this set and restore, and removes the key afterwards where it was absent, while L673 leaves it set to "5". Likewise, the metadata at L739-742 is `StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7")`, which `ParquetSchemaSuite` already uses, and then the `ResolveDefaultColumns` import could go. Could these use them?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
+ try {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ rethrow(under(2, oom), fromRead = false)
+ } finally {
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, depth.toString)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ rethrow(new InternalError("codec"), fromRead = true)
+ rethrow(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ rethrow(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ rethrow(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val withDefault = new MetadataBuilder()
+ .putString(ResolveDefaultColumns.EXISTS_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .putString(ResolveDefaultColumns.CURRENT_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .build()
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = true, withDefault), StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split") {
Review Comment:
Minor: this also passes if the reader disarmed the filter for the rest of the split after the first error. Both predicates are always true, so `result == rows` holds either way, and the second row group's error is then never reached, so `warnings == 1` holds too. No test pins that an evaluation or decoding give-up affects only its own row group: the other error tests read one row group, and L1837 covers the budget path. Could this use three row groups with a selective predicate on an error-free middle one, and assert that its rows are filtered?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
+ try {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ rethrow(under(2, oom), fromRead = false)
+ } finally {
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, depth.toString)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ rethrow(new InternalError("codec"), fromRead = true)
+ rethrow(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ rethrow(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ rethrow(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val withDefault = new MetadataBuilder()
+ .putString(ResolveDefaultColumns.EXISTS_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .putString(ResolveDefaultColumns.CURRENT_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .build()
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = true, withDefault), StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split") {
+ // Two row groups give the filter up on an evaluation error, and the reader reports that once.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val rowGroups = footerOf(path).getBlocks.asScala.map(_.getRowCount)
+ assert(rowGroups.size == 2 && rowGroups.head >= 5L && rowGroups.head < 150L,
+ s"keys 5 and 150 must fall in two row groups; got $rowGroups")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val filter = createFilter(Seq(And(failingAt(5L), failingAt(150L))), kvSchema())
+ val ((result, reader), warnings) =
+ withLogged("applying it to a row raised an error")(readAll(path, filter))
+ try {
+ assert(result == rows, s"both row groups must come back whole; got ${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ val (_, sevenUncapped, _) = run("64MB", 394L)
+ val (_, sevenAtCharge, _) = run("103b", 394L)
+ val (_, sevenUnderCharge, _) = run("102b", 394L)
+ assert(sevenAtCharge == sevenUncapped,
+ s"a cap of exactly the charge must keep splicing; got $sevenAtCharge against " +
+ sevenUncapped)
+ assert(sevenUnderCharge > sevenUncapped,
+ s"a cap one byte under it must give splicing up; got $sevenUnderCharge against " +
+ sevenUncapped)
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the rows left and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the rows that can still reach it. That memory is taken before any value
+ // arrives, so it is charged then. Six survivors over a capacity of 4 fill one set of four
+ // 1000-byte keys, and the second set starts two rows before the end of the row group. So it
+ // is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000 bytes
+ // each, so they add 8000 more once the sizing is used up, and the row group holds 32142 bytes
+ // in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or a
+ // double charge would each get wrong. The second set's byte child is reserved for 2000 bytes,
+ // which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000. And
+ // the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)
+
+ val (uncappedRows, uncappedBytes) = run("64MB")
Review Comment:
Nit: with this round, the check that a cap of exactly the charge splices and one byte under gives up is written three times, here, at L1255-1262 and at L1066-1074, each with the same three reads and two assertions. Could one helper, e.g. `assertSpliceCharge(charge, rows)(read)`, serve all three? Two smaller ones from the same round: `withPartialReadWarning` (L228-231) now only turns `withLogged`'s count back into a Boolean, while every other check asserts the count, and `readWith` (L599) has one caller left.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
+ try {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ rethrow(under(2, oom), fromRead = false)
+ } finally {
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, depth.toString)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ rethrow(new InternalError("codec"), fromRead = true)
+ rethrow(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ rethrow(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ rethrow(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val withDefault = new MetadataBuilder()
+ .putString(ResolveDefaultColumns.EXISTS_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .putString(ResolveDefaultColumns.CURRENT_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .build()
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = true, withDefault), StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split") {
+ // Two row groups give the filter up on an evaluation error, and the reader reports that once.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val rowGroups = footerOf(path).getBlocks.asScala.map(_.getRowCount)
+ assert(rowGroups.size == 2 && rowGroups.head >= 5L && rowGroups.head < 150L,
+ s"keys 5 and 150 must fall in two row groups; got $rowGroups")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val filter = createFilter(Seq(And(failingAt(5L), failingAt(150L))), kvSchema())
+ val ((result, reader), warnings) =
+ withLogged("applying it to a row raised an error")(readAll(path, filter))
+ try {
+ assert(result == rows, s"both row groups must come back whole; got ${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ val (_, sevenUncapped, _) = run("64MB", 394L)
+ val (_, sevenAtCharge, _) = run("103b", 394L)
+ val (_, sevenUnderCharge, _) = run("102b", 394L)
+ assert(sevenAtCharge == sevenUncapped,
+ s"a cap of exactly the charge must keep splicing; got $sevenAtCharge against " +
+ sevenUncapped)
+ assert(sevenUnderCharge > sevenUncapped,
+ s"a cap one byte under it must give splicing up; got $sevenUnderCharge against " +
+ sevenUncapped)
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the rows left and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the rows that can still reach it. That memory is taken before any value
+ // arrives, so it is charged then. Six survivors over a capacity of 4 fill one set of four
+ // 1000-byte keys, and the second set starts two rows before the end of the row group. So it
+ // is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000 bytes
+ // each, so they add 8000 more once the sizing is used up, and the row group holds 32142 bytes
+ // in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or a
+ // double charge would each get wrong. The second set's byte child is reserved for 2000 bytes,
+ // which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000. And
+ // the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)
+
+ val (uncappedRows, uncappedBytes) = run("64MB")
+ val (atChargeRows, atChargeBytes) = run("32142b")
+ assert(uncappedRows == 6 && atChargeRows == 6,
+ s"the filter keeps keys 15..20 either way; got $uncappedRows and $atChargeRows")
+ assert(atChargeBytes == uncappedBytes,
+ "a cap of exactly the row group's charge must keep splicing and read the key column " +
+ s"once; got $atChargeBytes against $uncappedBytes")
+ val (_, underChargeBytes) = run("32141b")
+ assert(underChargeBytes > uncappedBytes,
+ "a cap one byte under it must give splicing up and read the key column again; " +
+ s"got $underChargeBytes against $uncappedBytes")
+
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ if (cap >= 32142L) {
+ assert(secondSet == Seq(4000),
+ s"the second set must be sized for the two values left under cap $cap; got $secondSet")
+ }
+ }
+ }
+ }
+
+ test("a later accumulator set is sized only when it fits with its rows' fixed part") {
+ // The second set's two keys are 500 bytes here, shorter than the first set's 1000, so growing
+ // value by value is the cheaper path: 20142 bytes in all against 24142 sized. A cap of 24142
+ // sizes the set, whose byte child is then reserved for 2000 bytes and holds 4000. One of
+ // 24141 leaves 8033 bytes after the first set and its range, a byte short of the 8000 of
+ // sizing and the 34 of the two rows' fixed part. So the set grows instead, to a 1000-byte
+ // child, and keeps splicing. A cap of 20141 gives splicing up.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 500)
+ def run(maxSplicedBytes: String): Long =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)._2
+
+ val uncapped = run("64MB")
+ assert(run("24142b") == uncapped, "a cap that fits the sizing must keep splicing")
+ assert(run("24141b") == uncapped, "and so must one that only fits growing value by value")
+ assert(run("20141b") > uncapped, "a cap under both must give splicing up")
+
+ Seq(Long.MaxValue -> 4000, 24142L -> 4000, 24141L -> 1000).foreach { case (cap, child) =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected, s"every survivor must come back with its own key under cap $cap")
+ assert(secondSet == Seq(child),
+ s"under cap $cap the second set's byte child must hold $child bytes; got $secondSet")
+ }
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, schema, storageFilters, maxSplicedBytes, batchSize = 16)
+ val (_, uncapped) = run("64MB")
+ val (rows, atCharge) = run("166b")
+ val (_, underCharge) = run("165b")
+ assert(rows == 7, s"the filter keeps keys 394..400; got $rows")
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Two row groups of 100 rows with `v = k * 10`, and a pushed `v IN (x, 1500)`, where `x` falls
+ // between the first two pages of the first row group. Its statistics keep that row group, while
+ // its column index keeps no page of it.
+ private def writeColumnIndexEmptiedFile(dir: File): (String, Seq[sources.Filter]) = {
Review Comment:
Nit: this and `valueBetweenFirstTwoPages` are used only by the tests at L3220 and L3238 in the "Page-level pushedFilterRanges" section, but they sit between `shrinkDictionary` and `overwriteBytes`, which only the file-damaging helpers call. Could they move under the L3098 header, next to their tests? `asTask` (L235) similarly splits the log-capture helpers and could sit beside `formatReader` and `readLongs`.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val (result, logged) = withLogged(partialReadWarning)(body)
+ (result, logged > 0)
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def readWith(key: Expression): SparkException = intercept[SparkException](read(key))
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(readWith(key))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged("applying it to a row raised an error") {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ rethrow(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ rethrow(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, "2")
+ try {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ rethrow(under(2, oom), fromRead = false)
+ } finally {
+ SparkEnv.get.conf.set(KILL_ON_FATAL_ERROR_DEPTH.key, depth.toString)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ rethrow(new InternalError("codec"), fromRead = true)
+ rethrow(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ rethrow(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ rethrow(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val withDefault = new MetadataBuilder()
+ .putString(ResolveDefaultColumns.EXISTS_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .putString(ResolveDefaultColumns.CURRENT_DEFAULT_COLUMN_METADATA_KEY, "7")
+ .build()
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = true, withDefault), StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split") {
+ // Two row groups give the filter up on an evaluation error, and the reader reports that once.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val rowGroups = footerOf(path).getBlocks.asScala.map(_.getRowCount)
+ assert(rowGroups.size == 2 && rowGroups.head >= 5L && rowGroups.head < 150L,
+ s"keys 5 and 150 must fall in two row groups; got $rowGroups")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val filter = createFilter(Seq(And(failingAt(5L), failingAt(150L))), kvSchema())
+ val ((result, reader), warnings) =
+ withLogged("applying it to a row raised an error")(readAll(path, filter))
+ try {
+ assert(result == rows, s"both row groups must come back whole; got ${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ val (_, sevenUncapped, _) = run("64MB", 394L)
+ val (_, sevenAtCharge, _) = run("103b", 394L)
+ val (_, sevenUnderCharge, _) = run("102b", 394L)
+ assert(sevenAtCharge == sevenUncapped,
+ s"a cap of exactly the charge must keep splicing; got $sevenAtCharge against " +
+ sevenUncapped)
+ assert(sevenUnderCharge > sevenUncapped,
+ s"a cap one byte under it must give splicing up; got $sevenUnderCharge against " +
+ sevenUncapped)
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the rows left and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the rows that can still reach it. That memory is taken before any value
+ // arrives, so it is charged then. Six survivors over a capacity of 4 fill one set of four
+ // 1000-byte keys, and the second set starts two rows before the end of the row group. So it
+ // is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000 bytes
+ // each, so they add 8000 more once the sizing is used up, and the row group holds 32142 bytes
+ // in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or a
+ // double charge would each get wrong. The second set's byte child is reserved for 2000 bytes,
+ // which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000. And
+ // the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)
+
+ val (uncappedRows, uncappedBytes) = run("64MB")
+ val (atChargeRows, atChargeBytes) = run("32142b")
+ assert(uncappedRows == 6 && atChargeRows == 6,
+ s"the filter keeps keys 15..20 either way; got $uncappedRows and $atChargeRows")
+ assert(atChargeBytes == uncappedBytes,
+ "a cap of exactly the row group's charge must keep splicing and read the key column " +
+ s"once; got $atChargeBytes against $uncappedBytes")
+ val (_, underChargeBytes) = run("32141b")
+ assert(underChargeBytes > uncappedBytes,
+ "a cap one byte under it must give splicing up and read the key column again; " +
+ s"got $underChargeBytes against $uncappedBytes")
+
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ if (cap >= 32142L) {
+ assert(secondSet == Seq(4000),
+ s"the second set must be sized for the two values left under cap $cap; got $secondSet")
+ }
+ }
+ }
+ }
+
+ test("a later accumulator set is sized only when it fits with its rows' fixed part") {
+ // The second set's two keys are 500 bytes here, shorter than the first set's 1000, so growing
+ // value by value is the cheaper path: 20142 bytes in all against 24142 sized. A cap of 24142
+ // sizes the set, whose byte child is then reserved for 2000 bytes and holds 4000. One of
+ // 24141 leaves 8033 bytes after the first set and its range, a byte short of the 8000 of
+ // sizing and the 34 of the two rows' fixed part. So the set grows instead, to a 1000-byte
+ // child, and keeps splicing. A cap of 20141 gives splicing up.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 500)
+ def run(maxSplicedBytes: String): Long =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)._2
+
+ val uncapped = run("64MB")
+ assert(run("24142b") == uncapped, "a cap that fits the sizing must keep splicing")
+ assert(run("24141b") == uncapped, "and so must one that only fits growing value by value")
+ assert(run("20141b") > uncapped, "a cap under both must give splicing up")
+
+ Seq(Long.MaxValue -> 4000, 24142L -> 4000, 24141L -> 1000).foreach { case (cap, child) =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected, s"every survivor must come back with its own key under cap $cap")
+ assert(secondSet == Seq(child),
+ s"under cap $cap the second set's byte child must hold $child bytes; got $secondSet")
+ }
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, schema, storageFilters, maxSplicedBytes, batchSize = 16)
+ val (_, uncapped) = run("64MB")
+ val (rows, atCharge) = run("166b")
+ val (_, underCharge) = run("165b")
+ assert(rows == 7, s"the filter keeps keys 394..400; got $rows")
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Two row groups of 100 rows with `v = k * 10`, and a pushed `v IN (x, 1500)`, where `x` falls
+ // between the first two pages of the first row group. Its statistics keep that row group, while
+ // its column index keeps no page of it.
+ private def writeColumnIndexEmptiedFile(dir: File): (String, Seq[sources.Filter]) = {
+ val blocks =
+ (0 until 2).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ (path, Seq(sources.In("v", Array[Any](valueBetweenFirstTwoPages(path), 1500L))))
+ }
+
+ // A value of the long column `v` that falls between the first two pages of the first row group.
+ private def valueBetweenFirstTwoPages(filePath: String): Long =
+ Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val index = reader.readColumnIndex(reader.getRowGroups.get(0).getColumns.get(1))
+ def asLong(bytes: java.nio.ByteBuffer): Long =
+ bytes.duplicate().order(java.nio.ByteOrder.LITTLE_ENDIAN).getLong(0)
+ val firstMax = asLong(index.getMaxValues.get(0))
+ val secondMin = asLong(index.getMinValues.get(1))
+ assert(secondMin - firstMax > 1,
+ s"no gap between the first two pages: $firstMax, $secondMin")
+ firstMax + 1
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize = 64L * 1024 * 1024,
+ options = Map(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT -> "100"))
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ // The rows a reader hands back before it throws, what it throws, and, for the filtering
+ // reader, whether its scratch vectors were replaced on the way.
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable, Boolean) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(false, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(false, 16, storageFilter)
+ }
+ def scratch: Seq[WritableColumnVector] = reader match {
+ case filtering: LateMaterializationParquetRecordReader =>
+ ParquetTestAccess.keyScratchVectors(filtering)
+ case _ => Nil
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val scratchBefore = scratch
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ (read.toSeq, error, scratch.zip(scratchBefore).exists { case (a, b) => a ne b })
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError, _) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError, scratchReplaced) =
+ readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ // A failed decode can leave the scratch vectors grown. Phase 2 reads the same pages into
+ // vectors of its own, so the reader replaces them with capacity-sized ones rather than keep
+ // the grown ones for the rest of the split.
+ assert(scratchReplaced, "the decoding fallback must replace the scratch vectors")
+
+ // A kill that shows only at the decoding catch, after the per-chunk check has passed, is
+ // seen there. Without that catch's own check the row group would fall back and phase 2
+ // would fail on the page instead.
+ val killedAtTheCatch = spy(TaskContext.empty())
+ doAnswer((_: InvocationOnMock) => Boolean.box(Thread.currentThread().getStackTrace
+ .exists(_.getMethodName.startsWith("rethrowIfMustPropagate"))))
+ .when(killedAtTheCatch).isInterrupted()
+ val (_, killed, _) = TaskContext.withTaskContext(killedAtTheCatch) {
+ readUntilFailure(keyAtLeastFilter(500L))
+ }
+ assert(killed.isInstanceOf[SparkException] && killed.getMessage.contains("task kill") &&
+ killed.getCause != null && killed.getCause.getClass == plainError.getClass,
+ s"the kill must go out from the decoding catch, around the decoding error; got $killed")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+
+ test("a key vector that cannot grow, or a codec's InternalError, gives the row group up") {
+ // Both are fatal-looking errors that decoding the key pages can raise, and the reader falls
+ // back on both. A scratch vector that cannot grow is phase 1's own, and phase 2 then reads
+ // what a plain read reads. An `InternalError` from decoding is how a codec like hadoop-lzo
+ // reports a corrupt page, which the plain read of the same pages meets again. Raised from the
+ // filter instead, the first goes out as its OutOfMemoryError and the second in an
+ // `UnknownError`.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ Seq(
+ new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve")),
+ new InternalError("codec")).foreach { error =>
+ val failing: KeyReaderHook.OnKeyReader = (_, keyReader) => {
+ val failingReader = spy(keyReader)
+ doThrow(error).when(failingReader).readBatch(anyInt(), any(), any(), any())
+ failingReader
+ }
+ val ((result, reader), warnings) = try {
+ withLogged("decoding its key columns raised an error") {
+ readAll(path, keyAtLeastFilter(10L), onKeyReader = failing)
+ }
+ } catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case e: VirtualMachineError => fail(s"the reader must fall back on $error; got $e", e)
+ }
+ try {
+ assert(result == rows, s"the row group must come back whole on $error; got $result")
+ assert(warnings == 1, s"the decoding fallback must report $error; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("a key value the filter does not read and cannot be decoded gives splicing up") {
+ // The filter need not read every key on every row, as `coalesce(a, b)` reads `b` only where
+ // `a` is null. The copy into the survivor buffer reads every key, so it can meet a value no
+ // plain read decodes. A dictionary-encoded column decodes a value only when it is read, so a
+ // dictionary that holds fewer values than its ids point at is such a value. Copying it gives
+ // splicing up for the row group, and phase 2 reads the key columns the way a plain read does.
+ // So the read answers where a plain one does, and the filter still applies.
+ withTempDir { dir =>
+ // Two copies of each `b`, so parquet keeps its dictionary. Cut to 64 values, it leaves `b`
+ // undecodable from row 128 on. `a` is null only before that, so the filter never reads a
+ // damaged `b`.
+ val rows = (0 until 200).map { i =>
+ (if (i < 100 && i % 5 == 0) null else s"a_$i", s"b_${i / 2}", i)
+ }
+ val path = writeSingleParquetFile(dir, rows.toDF("a", "b", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE, dictionary = true)
+ assert(encodingsOf(path, "b").exists(_.usesDictionary),
+ s"the fixture's whole point; got ${encodingsOf(path, "b")}")
+ shrinkDictionary(path, column = "b", entries = 64)
+ val columns = Seq("a", "b", "v")
+ intercept[ArrayIndexOutOfBoundsException] {
+ readAllWith(path, columns, null, (b, i) => b.column(1).getUTF8String(i).toString)._2
+ .close()
+ }
+
+ val schema = StructType(Seq(
+ StructField("a", StringType), StructField("b", StringType), StructField("v", IntegerType)))
+ val coalesced = Coalesce(Seq(BoundReference(0, StringType, nullable = true),
+ BoundReference(1, StringType, nullable = true)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(coalesced, Literal.create("a_15", StringType))), schema)
+ val ((result, reader), givenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, columns, filter, (b, i) => b.column(2).getInt(i), capacity = 16)
+ }
+ try {
+ // Keeps rows 150-199 among others, whose `b` is damaged.
+ val expected = rows.filter { case (a, b, _) => Option(a).getOrElse(b) >= "a_15" }.map(_._3)
+ assert(expected.contains(150), "the filter must keep a row whose `b` is damaged")
+ assert(result == expected, s"got ${result.size} rows, expected ${expected.size}")
+ assert(givenUp == 1,
+ "the copy must have failed and given splicing up once, or this proves nothing; " +
+ s"logged $givenUp")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry reads the damaged row group whole over the pushed data filter's ranges, and only
+ // that one, since parquet keeps a column index store per block. What it must not give up is
+ // the pushed data filter's own page pruning, which a plain read of the same query keeps, nor
+ // the other row groups' narrowing. Dropping either would read more than it needs to.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warned) = withPartialReadWarning(read(evenKeys))
+ assert(warned, "the read must have discovered the damaged index, or this test proves nothing")
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ // The row groups around the damaged one keep the storage filter's narrowing too, so they
+ // return even keys alone.
+ val undamaged = filtered.filter(k => k <= 100L || k > 200L)
+ assert(undamaged.nonEmpty && undamaged.forall(_ % 2 == 0),
+ s"the other row groups must still be narrowed; got ${undamaged.filter(_ % 2 != 0)}")
+
+ // The WARN is reported once per split, however many row groups it costs.
+ val twiceDamaged = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(twiceDamaged, blockIdx = 0, columnIdx = 0)
+ breakOffsetIndex(twiceDamaged, blockIdx = 1, columnIdx = 0)
+ val (_, warnings) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, evenKeys), twiceDamaged, ordinal = 0)
+ }
+ assert(warnings == 1, s"two damaged row groups must be reported once; got $warnings")
+ }
+ }
+
+ test("a kill that lands before an unreadable offset index is reported as a kill") {
+ // Parquet turns any IOException reading an offset index into `MissingOffsetIndexException`,
+ // a kill's interrupt included, so the catch asks for a kill before it takes a missing index.
+ // Here the kill lands while the last row group's one chunk is evaluated, after the chunk's own
+ // check, and with no row group after it. So only that catch can see it.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 2, columnIdx = 0)
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val killsAt202 = EqualTo(Remainder(
+ OnKey(BoundReference(0, LongType, nullable = false), at = 202L, killsTask = true),
+ Literal(2L)), Literal(0L))
+ val (thrown, warned) = withPartialReadWarning {
+ asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ readLongs(formatReader(schema, pushed, Seq(killsAt202)), path, ordinal = 0)
+ }
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException], s"got $thrown")
+ assert(!warned, "the kill must not be logged as an unreadable offset index")
+ }
+ }
+
+ test("a v2 file's DELTA_BYTE_ARRAY decimal column is read over scattered survivors") {
+ // Phase 2 skips rows inside a page whenever survivors scatter, where a plain read skips only
+ // when a pushed filter cuts a page. The parquet v2 writer stores a FIXED_LEN_BYTE_ARRAY
+ // decimal without a dictionary as DELTA_BYTE_ARRAY, whose reader could not skip until
+ // SPARK-59828, so this read failed where a plain read of the same file did not.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 1000).selectExpr("id AS k", "CAST(id * 1.25 AS DECIMAL(30, 2)) AS d"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.WRITER_VERSION -> "PARQUET_2_0"))
+ assert(encodingsOf(path, "d").contains(Encoding.DELTA_BYTE_ARRAY),
+ s"the fixture's whole point; got ${encodingsOf(path, "d")}")
+ val everyThirdRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(3L)), Literal(0L))
+ val schema = kvSchema(valueType = DecimalType(30, 2))
+ val (result, reader) = readAllWith(path, Seq("k", "d"),
+ createFilter(Seq(everyThirdRow), schema),
+ (b, i) => (b.column(0).getLong(i), b.column(1).getDecimal(i, 30, 2).toJavaBigDecimal))
+ try {
+ val expected = (0L until 1000L by 3L)
+ .map(k => (k, new java.math.BigDecimal(k).multiply(new java.math.BigDecimal("1.25"))
+ .setScale(2)))
+ assert(result == expected, s"got ${result.take(5)} (${result.size} rows)")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the survivor queue and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ val fileSchema = kvSchema()
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAll(path, filter, capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // Each survivor costs 9 bytes of long key and 40 of its own range. The sum passes the cap of
+ // 1024 at survivor 21, which releases the buffer, and the ranges alone at survivor 26, which
+ // gives the filter up. The capacity decides what the buffer holds when it is released. At
+ // 4096 that is one partly filled set of accumulators, and at 16 a full set in the queue as
+ // well. Both have to end with the queue empty and the filter given up. Keeping one without
+ // the other would leave phase 2 reading non-key columns while emit spliced keys from a queue
+ // holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAll(path, filter)
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSeq == Seq("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does, since the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers conjuncts under the relation's own column names") {
+ // The analyzer spells a reference the way the query did, so under case-insensitive resolution
+ // a bloom can name `_TMP_METADATA_ROW_INDEX` while the relation, and the schema the reader
+ // requests, call the column `_tmp_metadata_row_index`. The planner normalizes what it offers to
+ // the relation's names, so the format's gate sees the name the reader will see and declines.
+ // Without that the gate passed it, `ParquetStorageFilter.create` rejected it on the driver, and
+ // a query that runs with the conf off failed with it on.
+ withTempDir { dir =>
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", s"id AS $rowIndexName", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", rowIndexName, "v")
+ .queryExecution.optimizedPlan
+ def column(name: String): Attribute =
+ relation.output.find(_.name == name).getOrElse(fail(s"no $name in the relation output"))
+
+ def offered(key: Attribute): Seq[Expression] = {
+ val logical = LogicalFilter(
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ scanOf(physical).storageFilters
+ }
+
+ // As a control, an ordinary key spelled in another case is offered, under its own name.
+ val upperK = offered(column("k").withName("K"))
+ assert(upperK.size == 1, s"a bloom over `K` must reach the scan; got $upperK")
+ assert(upperK.head.references.map(_.name).toSet == Set("k"),
+ s"and it must carry the relation's name for the column; got ${upperK.head}")
+
+ val upperRowIndex =
+ offered(column(rowIndexName).withName(rowIndexName.toUpperCase(Locale.ROOT)))
+ assert(upperRowIndex.isEmpty,
+ s"a bloom over the row-index column must not be offered in any case; got $upperRowIndex")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for. A dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`, since `/` is floating-point division in Spark SQL, which would hand every
+ // key expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize, dictionary = dictionary)
+ }
+
+ // Every column chunk of the file, from its footer. The facts the tests below assert about a
+ // fixture are read off these rather than assumed from the write options.
+ private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] =
+ footerOf(filePath).getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq
+
+ private def footerOf(filePath: String): ParquetMetadata = {
+ val conf = spark.sessionState.newHadoopConf()
+ ParquetFooterReader.readFooter(
+ HadoopInputFile.fromPath(new Path(filePath), conf), ParquetMetadataConverter.NO_FILTER)
+ }
+
+ // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String, column: String = "k"): Set[Encoding] =
+ footerChunks(filePath).filter(_.getPath.toDotString == column)
+ .flatMap(_.getEncodings.asScala).toSet
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // builds the plain vectorized reader rather than the late-materialization one. Every read helper
+ // in this suite goes through here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need. The caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction. `onKeyReader`, if given, sees every key column reader phase 1 opens, see
+ // `KeyReaderHook`.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ if (storageFilter == null) {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } else if (onKeyReader != null) {
+ new KeyReaderHook(useOffHeap, capacity, storageFilter, onKeyReader)
+ } else {
+ new LateMaterializationParquetRecordReader(useOffHeap, capacity, storageFilter)
+ }
+ } { reader =>
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // A key value as a string in its *internal* representation, so the splicing path and the plain
+ // path can be compared without knowing the type and without external type conversion.
+ // `InternalRow.getAccessor` does the per-type read, which is why this takes the batch's row
+ // rather than the column vector.
+ private def renderValue(row: InternalRow, ordinal: Int, dt: DataType): String =
+ InternalRow.getAccessor(dt)(row, ordinal) match {
+ case null => "null"
+ // Two equal arrays are two objects, so the default toString would compare identities.
+ case bytes: Array[Byte] => bytes.mkString(",")
+ case value => value.toString
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path. Whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.getRow(i), 0, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage, one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why. There is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ // The byte-array copy and a later set's up-front sizing go through `putByteArray`, `getBinary`
+ // and `reserve`, which off-heap vectors implement apart. So the variable-length types run off
+ // heap too, over a capacity of 4, which gives the string and binary keys a later set to size.
+ offHeap <- if (ValueCopier.isVariableLength(dt)) Seq(false, true) else Seq(false)
+ } {
+ val encoding = (if (dictionary) "dictionary-encoded" else "plain-encoded") +
+ (if (offHeap) ", off-heap" else "")
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ // A copier that fails gives splicing up, and phase 2 then reads the keys plainly, so the
+ // rows alone would not show it.
+ val ((result, reader), splicingGivenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt),
+ capacity = if (offHeap) 4 else 4096, useOffHeap = offHeap)
+ }
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ assert(splicingGivenUp == 0, s"every $name survivor must be copied into the buffer")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // This is a regression test. TimeType is an AtomicType that passes every planning-time gate
+ // (it is batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a
+ // TIME join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // What FileSourceStrategy offers, what ParquetStorageFilter.create admits and what the reader
+ // can copy are one list, because isSupportedKeyType answers from ValueCopier. Both are asserted
+ // here. A type the copier has no case for would be a task failure rather than a planning-time
+ // rejection, and the delegation is what rules that out.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ // Throws for a type it has no copier for.
+ ValueCopier.forType(dt)
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ val e = intercept[SparkException](ValueCopier.forType(dt))
+ assert(e.isInternalError && !DataSourceUtils.shouldIgnoreCorruptFileException(e),
+ s"an unsupported key type is the reader's bug, which must fail loudly; got $e")
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("two storage filters on two key columns are both applied") {
+ // The filters are ANDed into one predicate over the key columns of both.
+ withTempDir { dir =>
+ val df = spark.range(0, 400).selectExpr(
+ "id AS k", "(id * 7) % 400 AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)),
+ LessThan(BoundReference(1, LongType, nullable = true), Literal(200L)))
+ val (result, reader) = readAllWith(path, Seq("k", "k2", "v"),
+ createFilter(storageFilters, schema),
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i), b.column(2).getUTF8String(i)
+ .toString), capacity = 16)
+ try {
+ val expected = (0L until 400L).map(i => (i, (i * 7) % 400, i.toString))
+ .filter { case (k, k2, _) => k >= 100L && k2 < 200L }
+ assert(result == expected, s"got ${result.size} rows, expected ${expected.size}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, which fixes the key row's
+ // layout. The reader pairs each key column's batch slot with its key-row position, so nothing
+ // depends on that order. Every other fixture mentions its keys in column order, and production
+ // does not promise that, since the conjuncts arrive in `afterScanFilters` order, which follows
+ // the join. So this has two key columns mentioned the other way round, and asserts both the
+ // sorted layout and that every key still comes back with its own row.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ keyAtLeast(100L))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // The emit path rewrites the key slots alone, and a partition slot is not one of them. It sits
+ // past the data columns and must keep the constant vector `initBatch` populated for it.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite. OffHeapColumnVector.close() frees
+ // the native buffer, so a read after close is a crash rather than stale data. A second close
+ // is harmless, since close() zeroes the addresses it freed. capacity = 16 over 71 survivors
+ // forces 5 emits, each closing the previous emit's dequeued key vectors. The other 29 rows
+ // are what makes a declining reader fail this.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = keyAtLeast(30L)
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (30L to 100L),
+ s"expected the 71 surviving keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // One ColumnarBatch is handed out for the whole read and its key slots are rewritten per
+ // batch, so a consumer holding on to an earlier getCurrentValue() would read the wrong
+ // vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new LateMaterializationParquetRecordReader(false, 8, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution, a column missing from the physical file -----
+
+ test("non-key column missing from a file: the evolved table reads and credits avoided bytes") {
+ // Schema evolution end to end, with the non-key column rather than the key missing from the
+ // older file. The clipped requested schema keeps the missing column, which the byte walks skip
+ // by themselves, since they go over the block's chunks and the older file has no chunk for it.
+ // What this pins is that such a read returns the right rows and still credits a saving.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // The older file has (k, v), the newer one (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSeq.sorted
+ val expected = (150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // A positive total says the byte walks ran over the older file's row groups as well.
+ // Asserting non-negativity would prove nothing. `SQLMetric.add` drops a negative and the
+ // value cannot go below zero.
+ val bytesRg = withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value + bytesPf.value > 0,
+ s"the walk must credit some avoided bytes; got rg=${bytesRg.value} pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = scanOf(spark.read.parquet(path).select("k", "v").queryExecution.executedPlan)
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Seq[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read. The rows come
+ // back sorted, so a row returned twice shows.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Seq[(Long, Option[Long])]) = {
+ val withSF = withStorageFilters(spark.sql(s"SELECT id, k FROM $table")) { attr =>
+ Seq(buildFilter(attr("k")))
+ }
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSeq.sorted
+ (withSF, collected)
+ }
+
+ test("all keys missing: an error on the constant reads the file rather than skipping it") {
+ // A file with none of the filter's key columns makes the predicate constant over it, so the
+ // reader decides once, from the values `initBatch` materialized for those columns. That
+ // evaluation can throw exactly as a row's value can. Here the older file's `k` reads as its
+ // DEFAULT, which is not a number, and the cast fails under ANSI. Failing open reads the file
+ // the way a plain scan would, so its rows survive. Skipping it would lose them, and rethrowing
+ // would fail a query a plain read answers.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true",
+ SQLConf.ANSI_ENABLED.key -> "true") {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql("ALTER TABLE evolved ADD COLUMN k STRING DEFAULT 'not-a-number'")
+ spark.sql("INSERT INTO evolved VALUES (4, '40'), (5, '50')")
+ val withSF = withStorageFilters(spark.sql("SELECT id, k FROM evolved")) { attr =>
+ Seq(GreaterThanOrEqual(Cast(attr("k"), LongType), Literal(1L)))
+ }
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val ids = collectRows(withSF).map(_.getLong(idPos)).toSeq.sorted
+ assert(ids == Seq(1L, 2L, 3L, 4L, 5L),
+ s"the older file must fail open and the newer one pass the cast; got $ids")
+ }
+ }
+ }
+
+ test("all keys missing: a checked exception on the constant reads the file") {
+ // The constant evaluation fails open under the same rule as a row's, a checked exception
+ // included. The older file's `k` reads as its DEFAULT 7, which the key expression throws on,
+ // so that file must be read the way a plain scan would. The newer file's keys evaluate
+ // normally and are rejected, since no Filter runs above this scan.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(OnKey(k, at = 7L, error = new SparkException(_)),
+ Literal(100L)))
+ assert(collected == Seq((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L))),
+ s"the older file must fail open and the newer one be filtered; got $collected")
+ }
+ }
+ }
+
+ test("a file with none of the projected non-key columns is read plainly") {
+ // The clipped requested schema keeps a column the file does not have, so a projected non-key
+ // column added after the older file was written still looks like something for phase 2 to
+ // read. Phase 2 reads nothing for it, so for that file the filter could only add cost, and the
+ // reader declines it there. With no Filter above this scan, the older file's rows come back
+ // unfiltered while the newer file's are still filtered.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTable("evolved_value") {
+ spark.sql("CREATE TABLE evolved_value (k BIGINT, v STRING) USING parquet")
+ spark.sql("INSERT INTO evolved_value VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+ spark.sql("ALTER TABLE evolved_value ADD COLUMN w STRING")
+ spark.sql("INSERT INTO evolved_value VALUES (4, 'd', 'x'), (5, 'e', 'y')")
+ val withSF =
+ scanWithStorageFilter(spark.sql("SELECT k, w FROM evolved_value"), threshold = 5L)
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val keys = collectRows(withSF).map(_.getLong(kPos)).toSeq.sorted
+ assert(keys == Seq(1L, 2L, 3L, 5L),
+ s"the older file must be read unfiltered and the newer one filtered; got $keys")
+ }
+ }
+ }
+
+ test("a file with none of a projected struct's requested fields is read plainly") {
+ // Under the legacy returnNullStructIfAllFieldsMissing, the clipped schema keeps a struct none
+ // of whose requested fields the file has, and adds no field it has, so phase 2 would read
+ // nothing for it. Only the struct's leaves are missing columns, not the struct, so the reader
+ // asks the file schema about every non-key leaf. The filter rejects every row, so a declined
+ // file comes back whole and one that is not comes back empty.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 20).selectExpr("id AS k", "named_struct('a', CAST(id AS INT)) AS s"),
+ rowGroupSize = 64 * 1024L)
+ val schema =
+ new StructType().add("k", LongType).add("s", new StructType().add("b", IntegerType))
+ val storageFilters = Seq(keyAtLeast(1000L))
+ withSQLConf(SQLConf.LEGACY_PARQUET_RETURN_NULL_STRUCT_IF_ALL_FIELDS_MISSING.key -> "true") {
+ val keys = readLongs(formatReader(schema, Nil, storageFilters), path, ordinal = 0)
+ assert(keys == (0L until 20L), s"the file must be read plainly; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Seq[(Long, Option[Long])] =
+ Seq((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test. The default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Seq[(Long, Option[Long])] = Seq((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected, s"got $collected; expected $expected")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(StorageFilterMetrics.ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Seq((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got $rejectsNull")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Seq[(Long, Option[Long])] =
+ Seq((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got $acceptsNull")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ val collected = collectRows(withSF).map(r => (r.getLong(0), r.getLong(2))).toSeq.sorted
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1))
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got ${collected.take(5)}")
+ }
+ }
+ }
+
+ test("repeated non-key columns line up with scattered survivors") {
+ // Phase 2 hands every leaf its ranges, so an array, a map and an array of structs are read
+ // with rows skipped inside their pages. A wrong skip would pair a key with another row's
+ // values, which the post-scan Filter cannot catch.
+ withTempDir { dir =>
+ val df = spark.range(0, 600).selectExpr("id AS k",
+ "sequence(0, CAST(id % 4 AS INT)) AS arr",
+ "map(CAST(id AS STRING), CAST(id AS INT)) AS m",
+ "array(named_struct('a', CAST(id AS INT)), named_struct('a', CAST(-id AS INT))) AS arrs")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(256L))
+ val toScala = CatalystTypeConverters.createToScalaConverter(df.schema)
+ def extract(b: ColumnarBatch, i: Int): Any = toScala(b.getRow(i))
+ val columns = df.schema.fieldNames.toSeq
+ val (all, plainReader) = readAllWith(path, columns, null, extract, capacity = 16)
+ plainReader.close()
+ val everyThirdRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(3L)), Literal(0L))
+ val (result, reader) = readAllWith(path, columns,
+ createFilter(Seq(everyThirdRow), df.schema), extract, capacity = 16)
+ try {
+ val expected = all.filter(_.asInstanceOf[Row].getLong(0) % 3 == 0)
+ assert(expected.size == 200, s"the plain read must see every row; got ${all.size}")
+ assert(result == expected, s"got ${result.take(3)}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF =
+ scanWithStorageFilter(spark.read.parquet(outDir).select("k", "s"), threshold = 195L)
+ val collected = collectRows(withSF)
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSeq.sorted
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k"))
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ test("a storage filter binds to the right column with partition and metadata columns in output") {
+ // `preparedStorageFilters` binds the filter's references against `output.take(requiredSchema
+ // .length)`, and `output` is the data columns, then the generated metadata columns, then the
+ // partition columns, then the constant metadata ones. A projection carrying all four is where a
+ // wrong prefix would bind the predicate to another column and drop rows the filter keeps, which
+ // the post-scan Filter cannot put back. The file's first column `a` is not projected, so the
+ // key sits at a different position in the relation's data schema than in the scan's.
+ withTempDir { dir =>
+ val outDir = new File(dir, "parted").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id * 7 AS a", "id AS k", "CONCAT('v_', id) AS v", "CAST(id % 3 AS INT) AS p")
+ .repartition(1)
+ .write
+ .partitionBy("p")
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select(
+ col("k"), col("v"), col("p"),
+ col("_metadata.row_index").as("ri"), col("_metadata.file_size").as("fs"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ // The assumption the binding rests on, stated where a change to `output` would break it.
+ assert(withSF.output.take(withSF.requiredSchema.length).map(_.name) ==
+ withSF.requiredSchema.fieldNames.toSeq,
+ s"the scan's leading attributes must be its data columns; got " +
+ s"${withSF.output.map(_.name)}")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val pPos = withSF.output.indexWhere(_.name == "p")
+ def collectAll(plan: SparkPlan): Seq[Seq[Any]] =
+ collectRows(plan).map(_.toSeq(plan.output.map(_.dataType))).toSeq
+ .sortBy(_(kPos).asInstanceOf[Long])
+ val filtered = collectAll(withSF)
+ // The same scan without the filter is the oracle, so the row indexes and file sizes come
+ // from the read rather than from arithmetic over how the partitions were written.
+ val expected = collectAll(withSF.copy(storageFilters = Nil))
+ .filter(_(kPos).asInstanceOf[Long] >= 150L)
+ assert(filtered == expected,
+ s"expected ${expected.size} rows with their own partition and metadata values; " +
+ s"got ${filtered.size}")
+ assert(filtered.map(_(pPos)).distinct.size == 3,
+ s"every partition value must still be represented; got ${filtered.map(_(pPos)).distinct}")
+ }
+ }
+ }
+
+ // ----- Planner gates -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false. It may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ //
+ // This hook answers about shapes and types alone. Whether the conjunct is deterministic and
+ // whether its references are projected columns of this scan are plan-visible, so they stay with
+ // the planner; `FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter`
+ // covers that half.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ // A cast key is supported. In ANSI mode it can throw on a row an earlier conjunct would have
+ // dropped, and the reader handles that where it arises, by giving the filter up for the row
+ // group. Declining here instead would also decline every widening cast, which is what type
+ // coercion inserts for a join between an int and a bigint column.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+
+ // The conf is read in the format rather than in the planner, and it is the per-scan question
+ // that carries it, asked once before anything per conjunct.
+ assert(!format.supportsStorageFilterPushdown(spark), "the feature is off with the conf off")
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ assert(format.supportsStorageFilterPushdown(spark), "and on with it on")
+ assert(!subclass.supportsStorageFilterPushdown(spark), "a subclass must not claim support")
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ assert(format.supportsStorageFilter(onCast),
+ "a cast key is pushed, and an evaluation error gives the row group up")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The rows are right either way, since the bloom stays in the post-scan Filter. What this gate
+ // saves is offering the scan a filter its reader cannot apply.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one. Every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSeq.sorted
+ assert(keys == (1L to 50L),
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support declines rather than delegating") {
+ // The default `FileFormat.buildReaderWithStorageFilters` answers None, and the caller falls
+ // back to the ordinary builder. That is what keeps the two builders from being able to call
+ // each other, and declining is safe because the post-scan Filter still holds the conjunct.
+ val storageFilters = Seq(keyAtLeast(1L))
+ val declined = new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration(), Map.empty)
+ assert(declined.isEmpty, "the default must not build a reader of its own")
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilterPushdown(spark),
+ "and it must not claim support either")
+ }
+
+ test("a scan whose format declines falls back to the ordinary reader and returns every row") {
+ // The check above is that the default answers None. This is the other half, that `inputRDD`
+ // then builds the ordinary reader and the scan still produces its rows. A `ParquetFileFormat`
+ // subclass is the shape that reaches it, since it inherits the override and declines by class,
+ // and its ordinary builder is the one an extension would have customized.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, threshold = 195L)
+ val declining = withSF.copy(
+ relation = withSF.relation.copy(fileFormat = new ParquetFileFormat() {})(spark))
+ val collected = executePlanCollect(declining).toSeq.sorted
+ assert(collected == rows,
+ s"a declining format reads the file plainly; got ${collected.size} of ${rows.size} rows")
+ }
+ }
+ }
+
+ Seq(false, true).foreach { aqe =>
+ test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") {
+ // AQE is on by default in production, and it is where the bloom subquery is planned by
+ // PlanAdaptiveSubqueries rather than PlanSubqueries, which is the path
+ // preparedStorageFilters' ScalarSubquery materialization depends on.
+ withBloomFilterTables {
+ val baseConf = Map(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString)
+ def run(pushdown: Boolean): (Int, Seq[(Long, Long)], Long) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, rows) = runBloomFilterJoin()
+ val excluded = rowsExcludedByStorageFilters(plan)
+ (countBloomFiltersInStorageFilters(plan),
+ rows.map(r => (r.getLong(0), r.getLong(1))).toSeq.sorted, excluded)
+ }
+ val (offFilters, off, _) = run(false)
+ val (onFilters, on, excluded) = run(true)
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ // Equal results are guaranteed by the post-scan Filter whether or not the scan applied the
+ // filter, so they alone would not notice AQE dropping it on the way to the final plan.
+ assert(offFilters == 0 && onFilters == 1,
+ s"the scan must carry the filter with the conf on and not with it off; " +
+ s"got on=$onFilters off=$offFilters")
+ // Nor would the filter count notice the reader declining it, which the metrics do.
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+
+ test("canonicalization keeps a storage-filter scan distinct from a plain one") {
+ // storageFilters is in doCanonicalize and in the case class equality, which is what stops
+ // exchange and subquery reuse from serving one scan's result to the other. Reuse compares
+ // canonicalized plans, so a scan that filters must not match a plain scan of the same file, and
+ // two scans carrying the same filter must still match.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, threshold = 25L)
+ val plain = withSF.copy(storageFilters = Nil)
+ assert(withSF != plain, "case class equality must take storageFilters into account")
+ assert(!withSF.sameResult(plain),
+ s"a filtering scan must not be reusable as a plain one:\n${withSF.canonicalized}\n" +
+ s"${plain.canonicalized}")
+ // A second, independently planned scan of the same file with the same filter still
+ // matches, so reuse is not disabled wholesale. Its key attribute carries a different
+ // exprId, which is what canonicalization normalizes away.
+ val sameSF = scanWithStorageFilter(path, threshold = 25L)
+ assert(withSF.sameResult(sameSF),
+ s"two scans with the same storage filter must stay reusable:\n" +
+ s"${withSF.canonicalized}\n${sameSF.canonicalized}")
+ val otherSF = scanWithStorageFilter(path, threshold = 30L)
+ assert(!withSF.sameResult(otherSF), "a different threshold is a different result")
+ }
+ }
+ }
+
+ test("whole-stage codegen off: the row-at-a-time path still applies the extracted bloom") {
+ // The planner's extraction gate is `fileFormat.supportBatch`, which does not look at
+ // whole-stage codegen, while `FileSourceScanExec.supportsColumnar` does. So with codegen off
+ // the bloom is still extracted, `returningBatch` is false, and the reader serves the spliced
+ // batch one row at a time. That is the only combination where the planner's gate is weaker
+ // than the runtime's.
+ withBloomFilterTables {
+ val baseConf = Map(
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "false")
+ def run(pushdown: Boolean): (Seq[(Long, Long)], Int, Boolean, Long) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, result) = runBloomFilterJoin()
+ val scans = collect(plan) { case s: FileSourceScanExec => s }
+ (result.map(r => (r.getLong(0), r.getLong(1))).toSeq.sorted,
+ countBloomFiltersInStorageFilters(plan),
+ scans.nonEmpty && scans.forall(!_.supportsColumnar),
+ rowsExcludedByStorageFilters(plan))
+ }
+ val (off, _, _, _) = run(false)
+ val (on, storageBlooms, noColumnarScan, excluded) = run(true)
+ assert(storageBlooms >= 1, s"the bloom must still be extracted with codegen off; " +
+ s"got $storageBlooms")
+ assert(noColumnarScan, "with codegen off no scan should output columnar batches")
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ // The assertions above hold whether or not the row path honored the filter, since the
+ // post-scan Filter answers the query either way. The metrics are what say it did.
+ assert(excluded > 0, s"the row path must have excluded rows; the metrics say $excluded")
+ }
+ }
+
+ test("off-heap: survivor, scratch and persistent vectors are freed") {
+ // A spliced batch's key vectors are freed at the next `nextBatch()`, and the reader's own
+ // vectors at `close()`. Freed off-heap memory shows as a zero address.
+ withTempDir { dir =>
+ val path = writeParquetFile(dir, (1L to 200L).map(i => (i, s"v_$i")), rowGroupSize = 256L)
+ val reader = new LateMaterializationParquetRecordReader(true, 16, keyAtLeastFilter(1L))
+ def address(vector: Any): Long =
+ vector.asInstanceOf[OffHeapColumnVector].valuesNativeAddress()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val persistentKeys = reader.resultBatch().column(0)
+ val scratch = ParquetTestAccess.keyScratchVectors(reader)
+ assert(reader.nextBatch())
+ val firstKeys = reader.resultBatch().column(0)
+ assert((firstKeys ne persistentKeys) && address(firstKeys) != 0,
+ "the first batch must be spliced, or this proves nothing")
+ assert(reader.nextBatch())
+ assert(address(firstKeys) == 0, "a spliced batch's keys must be freed by the next batch")
+ reader.close()
Review Comment:
Minor: this closes the reader mid-read, which is when `close()` has to free the survivor sets still published or queued. After the second `nextBatch()`, set 2 is in `publishedKeyVectors` and five more sets are queued. But L3024 checks a batch that `nextBatch()` freed, and L3026-3027 check the persistent and scratch vectors, so removing `closeSplicingState()` from `close()` (`LateMaterializationParquetRecordReader.java:271-272`) would leak those six off-heap sets and still pass. Nothing else would catch it: `Platform.freeMemory` is outside `TaskMemoryManager`'s leak check, and the LIMIT tests run on heap. Could this keep `reader.resultBatch().column(0)` after L3023 and assert its address is 0 after `close()`? The queued sets would need a `ParquetTestAccess` accessor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4174360132
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1051 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages {@link #reportOnce} has already reported. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /** Whether the reader has reported an offset index parquet could not read. */
+ private boolean reportedUnreadableOffsetIndex;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (!reportedUnreadableOffsetIndex) {
+ reportedUnreadableOffsetIndex = true;
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn("Reading a row group of {} without page-level storage filtering, because "
+ + "parquet could not read the offset index of a column the read needs ({}). The "
+ + "other row groups are filtered as usual. Reported once per split",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(t);
+ reportOnce(t, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
Review Comment:
Thanks, the scratch vectors kept what the failed decode grew them to. Any decoding fallback now releases them and allocates capacity-sized ones, as you suggest. The corrupt-key-page test asserts they are replaced.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while evaluating a storage filter, if it must not fall back
+ * on it by giving the filter up. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped, see [[throwIfKilled]]. The error may be the kill's
+ * interrupt surfacing through a UDF that wrapped it. `PythonRunner` reads a kill off the task
+ * context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had the post-scan Filter raised `e`, which it raises as it is, goes out
+ * as itself, so the executor finds it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in a checked wrapper. `FileScanRDD` would take a bare
+ * one for a corrupt file under `ignoreCorruptFiles`, where a plain plan raises it from the
+ * post-scan Filter, outside that catch. The executor finds it under the wrapper and the
+ * FAILED_READ_FILE around that at a `spark.executor.killOnFatalError.depth` of 3 or more, the
+ * default being 5.
+ *
+ * Anything else is the filter's own error, which the post-scan Filter raises again for a row it
+ * does evaluate, so the reader falls back on it.
+ */
+ def rethrowIfMustPropagate(e: Throwable): Unit = rethrowIfMustPropagate(e, fromRead = false)
+
+ /**
+ * As [[rethrowIfMustPropagate]], for an error decoding the key pages, with three differences:
+ * - a plain read raises that from the reader, where `FileScanRDD` wraps it in FAILED_READ_FILE
+ * before the executor looks, so a fatal error deeper in the chain has to sit one level
+ * shallower to count.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ * The reader falls back on those, and the plain read of the same pages meets the corrupt data
+ * again, or allocates what a plain read allocates.
+ */
+ def rethrowIfMustPropagateFromRead(e: Throwable): Unit =
+ rethrowIfMustPropagate(e, fromRead = true)
+
+ private def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new SparkException("A storage filter met a fatal error it must not absorb", internal)
Review Comment:
Done as you suggest. An `InternalError` from the filter goes out in an `UnknownError` now, with the `InternalError` as its cause. Neither the `ignoreCorruptFiles` catch nor the executor's kill case takes it, and the executor finds it at the top, so the depth caveat is gone. The policy test covers it with a kill pending too.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1051 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages {@link #reportOnce} has already reported. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /** Whether the reader has reported an offset index parquet could not read. */
+ private boolean reportedUnreadableOffsetIndex;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (!reportedUnreadableOffsetIndex) {
+ reportedUnreadableOffsetIndex = true;
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn("Reading a row group of {} without page-level storage filtering, because "
+ + "parquet could not read the offset index of a column the read needs ({}). The "
+ + "other row groups are filtered as usual. Reported once per split",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(t);
Review Comment:
All three catches go through `giveUpOnError` again, which takes `fromRead` and picks the rethrow. The corrupt-key-page test now also reads under a spied `TaskContext` whose `isInterrupted()` answers true only while the rethrow asks, so only the decoding catch can see the kill. Without that catch's rethrow the test fails.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while evaluating a storage filter, if it must not fall back
+ * on it by giving the filter up. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped, see [[throwIfKilled]]. The error may be the kill's
+ * interrupt surfacing through a UDF that wrapped it. `PythonRunner` reads a kill off the task
+ * context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had the post-scan Filter raised `e`, which it raises as it is, goes out
+ * as itself, so the executor finds it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in a checked wrapper. `FileScanRDD` would take a bare
+ * one for a corrupt file under `ignoreCorruptFiles`, where a plain plan raises it from the
+ * post-scan Filter, outside that catch. The executor finds it under the wrapper and the
+ * FAILED_READ_FILE around that at a `spark.executor.killOnFatalError.depth` of 3 or more, the
+ * default being 5.
+ *
+ * Anything else is the filter's own error, which the post-scan Filter raises again for a row it
+ * does evaluate, so the reader falls back on it.
+ */
+ def rethrowIfMustPropagate(e: Throwable): Unit = rethrowIfMustPropagate(e, fromRead = false)
+
+ /**
+ * As [[rethrowIfMustPropagate]], for an error decoding the key pages, with three differences:
+ * - a plain read raises that from the reader, where `FileScanRDD` wraps it in FAILED_READ_FILE
+ * before the executor looks, so a fatal error deeper in the chain has to sit one level
+ * shallower to count.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ * The reader falls back on those, and the plain read of the same pages meets the corrupt data
+ * again, or allocates what a plain read allocates.
+ */
+ def rethrowIfMustPropagateFromRead(e: Throwable): Unit =
+ rethrowIfMustPropagate(e, fromRead = true)
+
+ private def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new SparkException("A storage filter met a fatal error it must not absorb", internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ if (Executor.isFatalError(e, if (fromRead) fatalErrorDepth - 1 else fatalErrorDepth)) {
+ // The one `Executor.isFatalError` found, which no SparkOutOfMemoryError can precede.
+ val fatal = Iterator.iterate(e)(_.getCause).find(Utils.isFatalError).get
+ if (!corrupt(fatal)) throw out(fatal)
+ }
+ }
+
+ /**
+ * Whether `e` is how `WritableColumnVector.reserve` reports a vector that cannot grow: a plain
+ * `RuntimeException`, caused by an `OutOfMemoryError` or by nothing, for a capacity past what a
+ * vector can hold.
+ */
+ private def isVectorGrowthFailure(e: Throwable): Boolean =
+ e.getClass == classOf[RuntimeException] &&
Review Comment:
Both done. The null arm is gone, since that exception falls back anyway. The test now takes the exception from a real `reserve(-1)` and asserts its class, so a change there fails it.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/StorageFilterPushdownBenchmark.scala:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import java.io.File
+
+import org.apache.spark.SparkConf
+import org.apache.spark.benchmark.Benchmark
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.BloomFilterMightContain
+import org.apache.spark.sql.execution.{FileSourceScanExec, SparkPlan}
+import org.apache.spark.sql.execution.datasources.parquet.StorageFilterMetrics
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Benchmark to measure storage-filter pushdown in the vectorized Parquet reader. Every case runs
+ * the same join, whose runtime bloom filter is the storage filter, with
+ * `spark.sql.parquet.storageFilterPushdown.enabled` off and on.
+ *
+ * The fact table is written in its join key's order, so which keys a dimension table selects
+ * decides where the surviving rows fall:
+ * - clustered keys put the survivors in a few pages, so the reader skips the other row groups
+ * whole and the other pages of the row group that holds them. These cases show what the
+ * feature saves. The bloom filter's false positives land anywhere, though, so the more keys
+ * it holds, the more pages hold a survivor;
+ * - scattered keys put survivors in every page, so the reader can skip nothing and decodes the
+ * non-key columns range by range. These cases show what the feature costs.
+ * After each case it prints what the storage filter excluded, which does not depend on the
+ * machine.
+ *
+ * To run this benchmark:
+ * {{{
+ * 1. without sbt: bin/spark-submit --class <this class>
+ * --jars <spark core test jar>,<spark catalyst test jar> <spark sql test jar>
+ * 2. build/sbt "sql/Test/runMain <this class>"
+ * 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/Test/runMain <this class>"
+ * Results will be written to "benchmarks/StorageFilterPushdownBenchmark-results.txt".
+ * }}}
+ */
+object StorageFilterPushdownBenchmark extends SqlBasedBenchmark {
+
+ override def getSparkSession: SparkSession = {
+ val conf = new SparkConf()
+ .setAppName(this.getClass.getSimpleName)
+ // Since `spark.master` always exists, overrides this value
+ .set("spark.master", "local[1]")
+ .setIfMissing("spark.driver.memory", "3g")
+ .setIfMissing("spark.executor.memory", "3g")
+ .set(SQLConf.SHUFFLE_PARTITIONS.key, "1")
+ .set("spark.ui.enabled", "false")
+
+ SparkSession.builder().config(conf).getOrCreate()
+ }
+
+ private val numRows = 4L * 1024 * 1024
+ private val payloadColumns = 8
+
+ // The clustered cases select a contiguous range of keys from the middle of the table.
+ private val clustered = Seq("0.1%" -> 0.001, "1%" -> 0.01, "10%" -> 0.1)
+ // The scattered cases select keys by their hash, so the survivors are spread over every page.
+ private val scattered = Seq("3%" -> 3, "25%" -> 25, "50%" -> 50)
+
+ private def prepareTables(dir: File): Unit = {
+ val factPath = new File(dir, "fact").getCanonicalPath
+ // The payload is hashed, so it neither compresses nor dictionary-encodes away, and the bytes a
+ // skipped page saves are real.
+ val payload = (0 until payloadColumns).map(i => s"xxhash64(id, $i) AS v$i")
+ spark.range(0, numRows, 1, 1).selectExpr("id AS k" +: payload: _*).write.parquet(factPath)
+ spark.read.parquet(factPath).createOrReplaceTempView("fact")
+
+ def writeDimension(name: String, predicate: String): Unit = {
+ val path = new File(dir, name).getCanonicalPath
+ // `sel_id` exists for the query's filter on the dimension side, which is what makes the
+ // optimizer build a bloom filter from it at all.
+ spark.range(0, numRows, 1, 1).where(predicate).selectExpr("id AS k", "id AS sel_id")
+ .write.parquet(path)
+ spark.read.parquet(path).createOrReplaceTempView(name)
+ }
+ clustered.foreach { case (label, fraction) =>
+ val start = numRows / 2
+ val end = start + (numRows * fraction).toLong
+ writeDimension(dimension("clustered", label), s"id >= $start AND id < $end")
+ }
+ scattered.foreach { case (label, percent) =>
+ writeDimension(dimension("scattered", label), s"pmod(xxhash64(id), 100) < $percent")
+ }
+ }
+
+ private def dimension(layout: String, label: String): String =
+ s"${layout}_${label.replace("%", "").replace(".", "_")}"
+
+ private def query(dimension: String): String =
+ s"SELECT f.* FROM fact f JOIN $dimension d ON f.k = d.k WHERE d.sel_id >= 0"
Review Comment:
Done, and the benchmark is reworked around it. Every case now also runs as a scan of the fact table alone, under the same bloom filter built once outside the timed body, as you suggest. The scaladoc and the description give both forms.
The cases are now the selectivities the feature is for, 0.1% to 5%, on two tables of about 100 MB. One has wide rows of a 2 kB value and the other narrow rows of eight longs, each in small pages and in default ones. The narrow table with scattered keys is where it costs. There the scan alone is 0.8x to 0.9x on JDK 17 at 0.5% and above. The wide table in small pages, the shape of the real workload in the description, gains 2.0x to 5.0x on the scan alone.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1051 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages {@link #reportOnce} has already reported. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /** Whether the reader has reported an offset index parquet could not read. */
+ private boolean reportedUnreadableOffsetIndex;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (!reportedUnreadableOffsetIndex) {
+ reportedUnreadableOffsetIndex = true;
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn("Reading a row group of {} without page-level storage filtering, because "
+ + "parquet could not read the offset index of a column the read needs ({}). The "
+ + "other row groups are filtered as usual. Reported once per split",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(t);
+ reportOnce(t, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = predicate.eval(keyScratchBatch.getRow(r));
+ } catch (Throwable t) {
+ // Fall back on anything but a task kill or a fatal error. The predicate runs here without
+ // the conjuncts before it in the plan, so it can throw on a row a plain scan never
+ // evaluates it on (see `ParquetStorageFilter.isSupportedStorageFilter`). Giving the
+ // filter up returns every row the pushed filter kept, and the post-scan Filter decides
+ // them in its own order, raising the same error for a row it does evaluate.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK exception
+ // (`timestamp_seconds` of a decimal) or an `AssertionError`, and a Hive UDF throws a
+ // checked SparkException. Rethrowing would fail a query a plain read answers, or drop
+ // the rest of the file silently under `ignoreCorruptFiles`.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_EVALUATION);
+ return null;
+ }
+ if (!survives) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ if (currentKeyAccumulators == null) {
+ splicedBytes += allocateKeyAccumulators(
+ cap - splicedBytes - rangeBytes - keyFixedBytesPerRow);
+ }
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // The buffer is this reader's own and optional, so failing to fill it gives splicing up
+ // rather than the filter, and the memory goes back at once. What fails here:
+ // - a survivor vector that cannot grow;
+ // - a key value the predicate did not read on this row, such as `b` in
+ // `coalesce(a, b)`, since the copy decodes every key;
+ // - a bug of ours, which only the WARN then shows.
+ // Phase 2 reads the key columns the way a plain read does, so a value that cannot be
+ // decoded fails the read only where a plain read fails.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together.
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, and returns the charge for the
+ * byte children it sized up front. The rest of the set is charged per row as survivors arrive.
+ * A variable-length key's byte child is reserved for what the set queued before it held, where
+ * that is more than its default allocation, so it does not grow one doubling at a time for every
+ * capacity's worth of survivors. {@code reserve} doubles what it is asked for, so with its null
+ * bytes the child takes the factor's four bytes per value byte. That happens only when the
+ * charge fits within {@code headroom}, what the cap leaves after this survivor's own fixed part.
+ * The memory is taken before any value arrives, so it is charged now, as if those values had
+ * arrived, and the values written into it use that charge up before they add to it.
+ */
+ private long allocateKeyAccumulators(long headroom) {
+ WritableColumnVector[] previous = survivorBatches.peekLast();
+ int[] sizeTo = new int[keyColumns.length];
+ long charge = 0L;
+ for (int i = 0; i < keyColumns.length; i++) {
+ // Only a child the default allocation would not already hold gains from being sized.
+ int valueBytes = previous != null && keyColumns[i].variableLength()
+ ? previous[i].arrayData().getElementsAppended() : 0;
+ if (valueBytes > capacity * DEFAULT_CHILD_BYTES_PER_ROW) {
+ sizeTo[i] = valueBytes;
+ charge += VARIABLE_LENGTH_BYTES_FACTOR * valueBytes;
+ }
+ }
+ // The memory is taken before the check after this survivor, so only when it fits what is left
+ // of the cap. A set that is not sized up front grows as the first one does, charged value by
+ // value, which also keeps a nearly empty last set from being charged as a full one.
+ boolean sizeUpFront = charge <= headroom;
Review Comment:
Done. A later set is sized for the rows that can still reach it, at most a set's capacity, and only when that and the fixed part of those rows fit what the cap leaves. So a last set is no longer charged as a full one, and the test's window where a larger cap read more is gone. The test is re-pinned. Its second set starts two rows before the end, its byte child is reserved for two values rather than four, and caps of 32142 and 32141 splice and give up.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1051 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages {@link #reportOnce} has already reported. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /** Whether the reader has reported an offset index parquet could not read. */
+ private boolean reportedUnreadableOffsetIndex;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
Review Comment:
I'd rather keep the one catch here. The shape is a file whose offset index is corrupt, where the rows are right and the cost is one phase 1 of that row group. Probing the cached store would be a second way to find the same thing, and only with a pushed filter, so the damaged-index test would need a second variant to keep reaching the catch. If corrupt offset indexes turn out to be common, the probe can go in then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4084280609
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -421,11 +552,313 @@ public boolean nextBatch() throws IOException {
return true;
}
+ /**
+ * Splicing emit path. Closes the previous emit's dequeued key vectors (releasing survivor memory
+ * incrementally), advances to the next row group if needed via {@link #checkEndOfRowGroup()},
+ * dequeues one survivor key vector per key column, drives non-key column readers for {@code num}
+ * rows, and assembles a fresh {@link ColumnarBatch} interleaving key (dequeued) and non-key
+ * (persistent value-vector) slots in the original projection order.
+ * The per-emit batch is a transient view over vectors owned elsewhere; see {@link #close()} and
+ * {@link #closeSplicingState()}.
+ */
+ private boolean nextBatchSplicing() throws IOException {
+ if (pendingCloseKeyVectors != null) {
+ for (WritableColumnVector v : pendingCloseKeyVectors) {
+ if (v != null) v.close();
+ }
+ pendingCloseKeyVectors = null;
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) continue;
+ columnVectors[i].reset();
+ }
+ // Match the eager path and zero the outgoing batch before the terminal checks below. Without
+ // this, a terminal call leaves `columnarBatch` pointing at the previous emit whose key slots
+ // were just closed above -- with off-heap vectors those buffers are already freed, so a
+ // consumer that read the batch after nextBatch() returned false would see freed memory.
+ if (columnarBatch != null) columnarBatch.setNumRows(0);
+ if (hitEndOfData) return false;
+ if (rowsReturned >= totalRowCount) return false;
+ checkEndOfRowGroup();
+ if (hitEndOfData) return false;
+
+ int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned);
+
+ WritableColumnVector[] dequeued = new WritableColumnVector[keyVectorQueues.length];
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ dequeued[i] = keyVectorQueues[i].removeFirst();
Review Comment:
Fixed, and the shape changed rather than the assignment moving: a survivor vector now stays owned by its queue while the batch is built on it, and the next emit removes and closes it. So there is no window where a vector is out of the queues and not yet reachable from a field, and `pendingCloseKeyVectors` is gone.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +929,450 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+
+ // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the
+ // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and
+ // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes
+ // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index
+ // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's
+ // ranges under the full schema, and with it off we do not call getRowRanges at all.
+ lateMatReader.setRequestedSchema(requestedSchema);
+ // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT
+ // options.useColumnIndexFilter(), so calling it unconditionally would keep applying
+ // column-index filtering after a user turned it off. That conf is the documented escape hatch
+ // for files whose column index is wrong, and trusting a wrong column index here would drop
+ // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no
+ // longer holds this predicate.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ if (pushedFilterRanges.rowCount() == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // Both byte metrics are always wired in production (FileSourceScanLike creates all five
+ // whenever storageFilters is non-empty), so this only skips the work on the test-only path
+ // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so
+ // there is nothing here to avoid on the production path.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ long baselineRows = pushedFilterRanges.rowCount();
+ long baselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema,
+ pushedFilterRanges)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyRequestedSchema);
+ long phase1Bytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema,
+ pushedFilterRanges)
+ : 0L;
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite "
+ + pushedFilterRanges.rowCount() + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+
+ if (finalRanges.rowCount() == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely. We still
+ // paid phase-1 to read the key column, so the bytes avoided vs a no-storage-filter read
+ // are baseline - phase1 (the non-key bytes the no-filter path would have read).
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(baselineBytes - phase1Bytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyRequestedSchema is null); emit reconstructs each
+ // batch from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyRequestedSchema == null) {
+ keptRows = finalRanges.rowCount();
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(nonKeyRequestedSchema);
+ // requireOffsetIndexesForPhase2() already established that every projected column of every
+ // row group has an offset index, so this page-filtering read cannot fail for want of one.
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRanges.rowCount()
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ phase2Bytes = bytesAvoidedPf != null
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, nonKeyRequestedSchema,
+ finalRanges)
+ : 0L;
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(baselineBytes - phase1Bytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the leaf columns of {@code schema} when it reads
+ * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are
+ * included, because both are read whenever any page of a chunk is read.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them. Every caller for a given block walks the same
+ * metadata, so a skipped column drops out of the baseline and the per-phase totals alike.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ MessageType schema,
+ RowRanges rowRanges) {
+ if (schema == null || rowRanges.rowCount() == 0 || schema.getColumns().isEmpty()) {
+ return 0L;
+ }
+ BlockMetaData block = reader.getRowGroups().get(blockIndex);
+ long blockRowCount = block.getRowCount();
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : block.getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ boolean wholeBlock = rowRanges.rowCount() == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : schema.getColumns()) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Reads all rows of the given key-only {@link PageReadStore} (which contains only rows in
+ * {@code pushedFilterRanges}) in capacity-sized chunks, evaluates the storage filter on each row,
+ * builds a {@link RowRanges} of surviving rows in original block-row coordinates, and appends
+ * survivor key values into the per-key-column accumulators ({@link #currentKeyAccumulators}).
+ * When an accumulator hits {@link #capacity}, it's pushed into {@link #keyVectorQueues} and a
+ * fresh one is allocated. After all rows have been examined, any partial trailing accumulator is
+ * pushed too.
+ *
+ * <p>The result is a subset of {@code pushedFilterRanges}: rows not in {@code
+ * pushedFilterRanges} were never read and are implicitly excluded.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ long keyRowsTotal = pushedFilterRanges.rowCount();
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ long remaining = keyRowsTotal;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ appendSurvivorRowToAccumulators(r);
+ }
+ }
+ remaining -= num;
+ }
+
+ finalizePartialAccumulators();
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
Review Comment:
Fixed: the array is assigned before the loop and `close()` frees the scratch vectors by walking it element-wise, so `keyScratchBatch` is no longer what reaches them.
The guard stays, and it cannot be reached in the state you describe: an allocation failure here propagates out of the reader, the task fails and `close()` runs, so nothing calls this again on a half-filled array.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -421,11 +552,313 @@ public boolean nextBatch() throws IOException {
return true;
}
+ /**
+ * Splicing emit path. Closes the previous emit's dequeued key vectors (releasing survivor memory
+ * incrementally), advances to the next row group if needed via {@link #checkEndOfRowGroup()},
+ * dequeues one survivor key vector per key column, drives non-key column readers for {@code num}
+ * rows, and assembles a fresh {@link ColumnarBatch} interleaving key (dequeued) and non-key
+ * (persistent value-vector) slots in the original projection order.
+ * The per-emit batch is a transient view over vectors owned elsewhere; see {@link #close()} and
+ * {@link #closeSplicingState()}.
+ */
+ private boolean nextBatchSplicing() throws IOException {
+ if (pendingCloseKeyVectors != null) {
+ for (WritableColumnVector v : pendingCloseKeyVectors) {
+ if (v != null) v.close();
+ }
+ pendingCloseKeyVectors = null;
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) continue;
+ columnVectors[i].reset();
+ }
+ // Match the eager path and zero the outgoing batch before the terminal checks below. Without
+ // this, a terminal call leaves `columnarBatch` pointing at the previous emit whose key slots
+ // were just closed above -- with off-heap vectors those buffers are already freed, so a
+ // consumer that read the batch after nextBatch() returned false would see freed memory.
+ if (columnarBatch != null) columnarBatch.setNumRows(0);
+ if (hitEndOfData) return false;
+ if (rowsReturned >= totalRowCount) return false;
+ checkEndOfRowGroup();
+ if (hitEndOfData) return false;
+
+ int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned);
+
+ WritableColumnVector[] dequeued = new WritableColumnVector[keyVectorQueues.length];
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ dequeued[i] = keyVectorQueues[i].removeFirst();
+ }
+
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) continue;
+ ParquetColumnVector cv = columnVectors[i];
+ for (ParquetColumnVector leafCv : cv.getLeaves()) {
+ VectorizedColumnReader columnReader = leafCv.getColumnReader();
+ if (columnReader != null) {
+ columnReader.readBatch(num, leafCv.getValueVector(),
+ leafCv.getRepetitionLevelVector(), leafCv.getDefinitionLevelVector());
+ }
+ }
+ cv.assemble();
+ }
+ if (rowIndexGenerator != null) {
+ // Row-index column is identified by name (ROW_INDEX_TEMPORARY_COLUMN_NAME), which is a
+ // synthetic metadata column never referenced by a storage filter, so its slot is a non-key
+ // slot with a persistent ParquetColumnVector.
+ rowIndexGenerator.populateRowIndex(columnVectors, num);
+ }
+
+ ColumnVector[] cols = new ColumnVector[persistentBatchColumns.length];
+ // This walks batch slots in ascending order while `keyIdx` walks the survivor queues in
+ // key-row-position order, so it pairs the k-th smallest key slot with key-row position k.
+ // That is only the identity because `ParquetStorageFilter.create` sorts `keyColumnIndices`
+ // ascending -- see the comment there. `isKeyTopLevel` marks which slots are keys but not their
+ // position in that list, so this loop cannot reconstruct the pairing on its own: if the list
+ // ever stops being sorted, key columns silently swap places in the output batch.
+ int keyIdx = 0;
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ if (i < isKeyTopLevel.length && isKeyTopLevel[i]) {
+ cols[i] = dequeued[keyIdx++];
+ } else {
+ cols[i] = persistentBatchColumns[i];
+ }
+ }
+ columnarBatch = new ColumnarBatch(cols);
Review Comment:
Took the second option, since documenting an exception to a public method's contract is the worse half of the trade. There is one `ColumnarBatch` for the whole read again, and the emit path rewrites its key slots in place -- `ColumnarBatch` holds the column array by reference, its staging row included, so writing a slot publishes it. Two allocations per batch go away with it, and the hoist-once pattern works on this path.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala:
##########
@@ -165,6 +166,47 @@ trait FileFormat {
}
}
+ /**
+ * Like [[buildReaderWithPartitionValues]] but additionally accepts a sequence of storage filters:
+ * Catalyst expressions that the storage layer may evaluate to drive value-column IO pruning based
+ * on key-column evaluation (e.g., late materialization with a runtime bloom filter).
+ *
+ * The default implementation delegates to [[buildReaderWithPartitionValues]] and accepts no
+ * storage filter at all. File formats that support storage-filter pushdown (e.g., Parquet) should
+ * override this method.
+ *
+ * A non-empty `storageFilters` here is a planner bug, so the default body rejects it rather than
+ * dropping it. The planner removes an extracted conjunct from the post-scan `Filter`, so a reader
+ * that ignores it returns rows the filter rejects. Every other layer of the feature fails loudly
+ * for the same reason. Unreachable today, because `FileSourceStrategy.extractStorageFilters` only
+ * extracts for `ParquetFileFormat` itself, but that keeps the invariant one method away from the
+ * code that relies on it.
+ *
+ * Scalar subqueries inside `storageFilters` are expected to have been materialized before this
+ * method is called, so that the returned reader can be safely serialized to executors.
+ *
+ * `storageFilterMetrics` is an optional map of SQL metrics the reader can update during execution
+ * (e.g. number of row groups skipped). The scan is expected to expose these metrics via its
+ * `metrics` field so they show up in the SQL UI.
+ */
+ def buildReaderWithStorageFilters(
+ sparkSession: SparkSession,
+ dataSchema: StructType,
+ partitionSchema: StructType,
+ requiredSchema: StructType,
+ filters: Seq[Filter],
+ storageFilters: Seq[Expression],
+ options: Map[String, String],
+ hadoopConf: Configuration,
+ storageFilterMetrics: Map[String, SQLMetric] = Map.empty
+ ): PartitionedFile => Iterator[InternalRow] = {
+ require(storageFilters.isEmpty,
+ s"${getClass.getSimpleName} does not support storage-filter pushdown, but was given " +
+ storageFilters.mkString("[", ", ", "]"))
+ buildReaderWithPartitionValues(
Review Comment:
Closed by routing both public builders through a private `buildParquetReader`, and the scaladoc on `buildReaderWithStorageFilters` now says an override must do the same, `super` included, since that call is virtual too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5821627153
Here is a summary of my review comments (https://github.com/apache/spark/pull/58895#pullrequestreview-5309775040) on `c255f88`. I didn't find a wrong-result issue under the default settings.
**Correctness**
1. The `parquet.filter.columnindex.enabled=false` escape hatch only covers phase 0. Phase 2 still selects pages and synchronizes rows through the offset index, so a file with a corrupt offset index can return non-key values from the wrong rows while the post-scan `Filter` passes them (key hash is correct).
2. `maxSplicedRowGroupBytes` is checked only when an accumulator fills `capacity` rows, and the post-phase-1 check ignores `splicedBytes`. Large variable-length keys can exceed the cap by a wide margin, and the conf doc ("holds no more than the plain read path does") is not accurate.
3. The key-slot/queue pairing in `publishSurvivorKeyVectors` silently depends on `ParquetStorageFilter.create` sorting the ordinals.
4. `FileFormat.supportsStorageFilter` receives canonicalized expressions because `ExpressionSet.filter` applies the predicate to `e.canonicalized` (undocumented contract for third-party formats).
**Performance**
5. The row-range budget is multiplied by the number of phase-2 leaf columns (per-reader copies in `ParquetReadState`). Wide projections give the filter up after phase 1 has done all its work, repeatedly for every row group, which is slower than the feature off. Sharing `RowRanges.getRanges()` would fix it at the root.
6. A missing offset index is detected only via `MissingOffsetIndexException` in phase 2, although it is known from the footer (`getOffsetIndexReference() == null`). This costs wasted work and a stack-trace WARN per split, loses whole-row-group skips afterwards, and gives up the whole file when only a key column lacks an offset index.
7. The materialized bloom filter is duplicated into the task binary and deserialized twice per task (about 16MiB extra heap per concurrent task with the default max size).
8. Because `Cast` is not in the `canEvaluateUnconditionally` whitelist, even never-failing widening casts (e.g., `int` joined with `bigint`) are never pushed.
**Design**
9. The correctness of the byte metrics and reads relies on the implicit ordering of Parquet's per-block `ColumnIndexStore` cache (`MISSING_INDEX_STORE.getOffsetIndex()` returns `null`, which would NPE inside Parquet if the ordering changes).
10. A Parquet-named conf and Parquet-specific metrics live in the format-agnostic `FileSourceStrategy` / `FileSourceScanLike`.
**Tests**
11. The `>= 0` metric assertions can never fail because `SQLMetric` ignores negative values.
12. The `MissingOffsetIndexException` path is untested, although such a file can be written with the `ParquetFileWriter.writeDataPage` overloads without a row count.
13. No AQE-on test verifies that the storage filter is actually applied (only result equality, which the post-scan `Filter` guarantees anyway).
**Cleanup**
14. Duplicated code (`closeSplicingState` vs `abandonSplicing`, phase-0 ranges in `recordFileSkipped`) and redundant give-up signaling (`filterGivenUp`, `spliceCurrentRowGroup`, `accumulate`, `null` return).
15. Stale references to the nonexistent `extractStorageFilters` and outdated test comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4133524917
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Asked this way rather than up front, from the footer. Parquet resolves the index over the
+ // paths current at its first lookup for the block, which without a pushed data filter is
+ // the non-key columns alone, so a footer walk over the projection is both stricter than the
+ // read and blind to an index that is claimed but unreadable. It is also where the throw
+ // costs least: it lands before any data page is read.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Reading {} without page-level storage filtering: reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs. Row groups the filter empties are still skipped whole", e,
+ MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ // The retry only avoids the same wall because a store missing one column's offset index
+ // reports no column index either, so these ranges cover the whole block and parquet reads
+ // it without consulting an index. That is three parquet internals deep, so it is checked:
+ // a release that changes any of them should fail here rather than throw from the read.
+ if (baselineRows != blockRowCount) {
+ throw new IllegalStateException(String.format(
+ "Cannot read row group %d of %s without an offset index: the pushed filter selects "
+ + "%d of %d rows, so a plain read of the block is not what it asks for",
+ blockIdx, lateMatReader.getFile(), baselineRows, blockRowCount));
+ }
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up: it read what a plain scan
+ // reads, so the saving is a certain zero.
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ long phase2Bytes = compressedBytesForRowRanges(blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = lateMatReader.getRowRanges(blockIdx);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private long compressedBytesForRowRanges(
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore =
+ wholeBlock ? null : lateMatReader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ // The store answers null, rather than throwing, for a path it was not built with. That can
+ // only mean the block's store was built while a narrower schema was requested than this
+ // walk asks about, an ordering bug rather than a property of the file, and the footer tells
+ // the two apart. It is reported rather than thrown: this walk only produces a counter, and
+ // `ignoreCorruptFiles` turns any exception from a reader into a silently truncated file, so
+ // a byte metric must not be able to change the answer.
+ if (chunk.getOffsetIndexReference() != null && !loggedMissingStoreEntry) {
+ loggedMissingStoreEntry = true;
+ LOG.warn("Undercounting the storage filter's avoided bytes for {}: column "
+ + path.toDotString() + " of row group " + blockIndex + " has an offset index the "
+ + "block's column index store was not built with",
+ MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ }
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below: the bytes buffered for
+ // splicing, and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = storageFilter.test(keyScratchBatch.getRow(r));
+ } catch (RuntimeException e) {
+ if (!storageFilter.isEvaluationError(e)) throw e;
Review Comment:
Fixed, and the narrowing is gone entirely. The catch is `catch (RuntimeException e)` now, and `isEvaluationError` is deleted.
Your `timestamp_seconds(decimal_col)` case is the one named in the comment above it, because an `ArithmeticException` out of `longValueExact` is exactly what a `SparkThrowable` test could not see. The all-keys-missing decision does the same and reads the file unfiltered when the constant evaluation throws. `supportsStorageFilter` no longer asks `canEvaluateUnconditionally` either, so a widening `Cast` key is pushed, which is what type coercion inserts for an int-to-bigint join, and an error from it gives the row group up.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Asked this way rather than up front, from the footer. Parquet resolves the index over the
+ // paths current at its first lookup for the block, which without a pushed data filter is
+ // the non-key columns alone, so a footer walk over the projection is both stricter than the
+ // read and blind to an index that is claimed but unreadable. It is also where the throw
+ // costs least: it lands before any data page is read.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
Review Comment:
Documented in the conf doc, in your terms. `spark.sql.parquet.storageFilterPushdown.enabled` now says that narrowing a row group to part of its rows trusts the page index, that a read with no pushed data filter never consults it, that a wrong index there can pair a row's key with another row's values, and that `parquet.filter.columnindex.enabled=false` is the escape hatch.
That conf also changed. It no longer turns the feature off, it leaves the filter with the row groups it empties, so the hatch costs the page-level saving and nothing more.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know
+ // pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ }
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ // Reading part of a row group needs the offset index, so on a file without one phase 2
+ // reads the whole row group instead. The rows the filter rejected are then emitted and
+ // the post-scan Filter drops them. What the filter still saves on such a file is the row
+ // groups it empties, which is decided above and needs no index at all.
+ if (fileHasNoOffsetIndex && finalRowCount < baselineRows) {
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Asked this way rather than up front, from the footer. Parquet resolves the index over the
+ // paths current at its first lookup for the block, which without a pushed data filter is
+ // the non-key columns alone, so a footer walk over the projection is both stricter than the
+ // read and blind to an index that is claimed but unreadable. It is also where the throw
+ // costs least: it lands before any data page is read.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Reading {} without page-level storage filtering: reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs. Row groups the filter empties are still skipped whole", e,
+ MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ // The retry only avoids the same wall because a store missing one column's offset index
+ // reports no column index either, so these ranges cover the whole block and parquet reads
+ // it without consulting an index. That is three parquet internals deep, so it is checked:
+ // a release that changes any of them should fail here rather than throw from the read.
+ if (baselineRows != blockRowCount) {
+ throw new IllegalStateException(String.format(
+ "Cannot read row group %d of %s without an offset index: the pushed filter selects "
+ + "%d of %d rows, so a plain read of the block is not what it asks for",
+ blockIdx, lateMatReader.getFile(), baselineRows, blockRowCount));
+ }
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up: it read what a plain scan
+ // reads, so the saving is a certain zero.
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ long phase2Bytes = compressedBytesForRowRanges(blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = lateMatReader.getRowRanges(blockIdx);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private long compressedBytesForRowRanges(
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore =
+ wholeBlock ? null : lateMatReader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ // The store answers null, rather than throwing, for a path it was not built with. That can
+ // only mean the block's store was built while a narrower schema was requested than this
+ // walk asks about, an ordering bug rather than a property of the file, and the footer tells
+ // the two apart. It is reported rather than thrown: this walk only produces a counter, and
+ // `ignoreCorruptFiles` turns any exception from a reader into a silently truncated file, so
+ // a byte metric must not be able to change the answer.
+ if (chunk.getOffsetIndexReference() != null && !loggedMissingStoreEntry) {
+ loggedMissingStoreEntry = true;
+ LOG.warn("Undercounting the storage filter's avoided bytes for {}: column "
+ + path.toDotString() + " of row group " + blockIndex + " has an offset index the "
+ + "block's column index store was not built with",
+ MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ }
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below: the bytes buffered for
+ // splicing, and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = storageFilter.test(keyScratchBatch.getRow(r));
+ } catch (RuntimeException e) {
+ if (!storageFilter.isEvaluationError(e)) throw e;
+ // Fail open. The predicate ran on a row that, in the plan, an earlier conjunct would
+ // have rejected before it, so a plain scan never evaluates it there. Giving the filter up
+ // for this row group puts every row the pushed filter kept back in the output, and the
+ // post-scan Filter then evaluates the conjuncts in their own order. Either an earlier one
+ // drops the row before this expression runs, or it does not and the query fails the way
+ // it would have without this feature.
+ logFilterGivenUpOnError(e);
+ giveUpFilter();
+ return null;
+ }
+ if (survives) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) splicedBytes += appendSurvivorRowToAccumulators(r);
+ // Both halves of what this row group retains grow per survivor, and either can cross
+ // the budget on its own, so they are weighed together here and nowhere else. The cheaper
+ // concession comes first: release the buffer, and give the filter up as well if the
+ // ranges alone still do not fit.
+ long rangeBytes = rowRangeStateBytes(survivorRangeCount);
+ if (splicedBytes + rangeBytes > cap) {
Review Comment:
The counting is fixed. A variable-length key is charged 17 bytes per row rather than 9, which is the offset, the length, the null byte, and the byte child's eager four bytes with a null byte for each of them. For your 10-byte key at capacity 4096 the charge is now above what the vectors hold, which is the side to be wrong on.
The eagerly allocated next accumulator is gone too. The push and the reallocation moved to the top of the append, so a row group whose survivors are a multiple of the capacity leaves no unused set behind. What is still uncounted is the first set of a row group, which is capacity-sized whatever the survivors turn out to be, so the floor is one set per key column instead of a multiple of the cap.
The `MemoryConsumer` route is in the description's follow-up list, with `abandonSplicing()` as the `spill()` you point at. I kept it out of this PR because the reader would have to hold a `TaskMemoryManager`, and because this buffer cannot spill, it can only be discarded, which is what the cap already does.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -391,15 +574,54 @@ public void enableReturningBatches() {
* Advances to the next batch of rows. Returns false if there are no more.
*/
public boolean nextBatch() throws IOException {
+ releasePublishedKeyVectors();
for (ParquetColumnVector vector : columnVectors) {
vector.reset();
}
+ // Zeroed before the terminal checks below, so a terminal call cannot leave a spliced batch
+ // pointing at the key vectors just released, which off heap is freed memory.
columnarBatch.setNumRows(0);
+ if (hitEndOfData) return false;
if (rowsReturned >= totalRowCount) return false;
checkEndOfRowGroup();
+ if (hitEndOfData) return false;
int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned);
- for (ParquetColumnVector cv : columnVectors) {
+ // A spliced row group takes its key slots from the survivor queues, so phase 2 skipped them.
+ // Everything else read every projected column: a plain read with no storage filter at all, or a
+ // row group that gave splicing up, whose batch needs its key slots put back to the persistent
+ // vectors a previous row group rewrote. `spliceCurrentRowGroup` stays false without a storage
+ // filter, so it answers for all three.
+ if (spliceCurrentRowGroup) {
+ publishSurvivorKeyVectors(num);
+ } else if (spliceBatchColumns != null) {
+ System.arraycopy(persistentBatchColumns, 0, spliceBatchColumns, 0, spliceBatchColumns.length);
+ }
+ readPersistentColumns(num, /* skipKeySlots= */ spliceCurrentRowGroup);
+ // If needed, compute row indexes within a file. The row-index column is identified by name
+ // (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references,
+ // so its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ if (rowIndexGenerator != null) {
Review Comment:
Fixed by rejecting it. `ParquetStorageFilter.isSupportedStorageFilter` answers false for a conjunct that references a column named `_tmp_metadata_row_index`, so the planner never offers one, and `create` has a `require` for the same thing in case anything else builds a filter.
The reason is yours: this reader writes row indexes over whatever that column holds, so its value would depend on how each row group was read. Both checks go through a new `ParquetRowIndexUtil.isRowIndexColumnName`, so one place knows the name.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +945,677 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says.
+ filterGivenUp = false;
+ // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns
+ // again along with everything else.
+ spliceCurrentRowGroup = !fileHasNoOffsetIndex;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
Review Comment:
You are right, and it is inherent to evaluating the filter before the row group's first batch. Phase 1 decodes every key page of the row group, so a corrupt page anywhere in it costs that whole row group, where a plain read would have returned the batches before it.
Under `ignoreCorruptFiles` the answer is a subset either way, and how big a subset already depends on where the batch boundaries fall, so the result shrinks rather than being wrong. I left the conf doc alone because it was shortened in the same round. Say if you want a line there.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:
##########
@@ -190,6 +192,53 @@ class ParquetFileFormat
filters: Seq[Filter],
options: Map[String, String],
hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow] = {
+ buildParquetReader(
+ sparkSession, dataSchema, partitionSchema, requiredSchema, filters, Nil, options, hadoopConf,
+ Map.empty)
+ }
+
+ /**
+ * The conf that turns this on is read here rather than in the planner, the way `supportBatch`
+ * reads its own confs, so a Parquet-named conf does not decide for a format that is not Parquet.
+ *
+ * Subclasses answer false on purpose, even though they inherit this reader: a subclass may
+ * customize reading by overriding `buildReaderWithPartitionValues`, and a scan with storage
+ * filters routes through `buildReaderWithStorageFilters` instead, which would silently bypass
+ * whatever the subclass does.
+ */
+ override def supportsStorageFilterPushdown(sparkSession: SparkSession): Boolean =
+ getSqlConf(sparkSession).parquetStorageFilterPushdownEnabled &&
+ getClass == classOf[ParquetFileFormat]
+
+ override def supportsStorageFilter(expr: Expression): Boolean =
+ ParquetStorageFilter.isSupportedStorageFilter(expr)
+
+ override def buildReaderWithStorageFilters(
Review Comment:
Done. `buildReaderWithStorageFilters` answers `None` for a subclass, for the reason you give, that this entry point is reachable without the planner having asked the per-scan question, and that `None` sends the caller to the builder a subclass overrides.
The fallback is tested end to end now as well. A scan whose relation carries a `ParquetFileFormat` subclass and a storage filter returns every row of the file through the ordinary reader, and the test fails if the guard is removed.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -409,23 +631,250 @@ public boolean nextBatch() throws IOException {
}
cv.assemble();
}
- // If needed, compute row indexes within a file.
- if (rowIndexGenerator != null) {
- rowIndexGenerator.populateRowIndex(columnVectors, num);
- }
+ }
- rowsReturned += num;
+ /** Publishes {@code num} rows as the current batch. */
+ private void finishBatch(int num) {
columnarBatch.setNumRows(num);
+ rowsReturned += num;
numBatched = num;
batchIdx = 0;
- return true;
+ }
+
+ /**
+ * Points the batch's key slots at one survivor key vector each, which is what publishes them: the
+ * same {@link ColumnarBatch} is handed out every time, over an array it holds by reference. The
+ * queues keep owning those vectors until the next batch releases them, so a phase-2 read that
+ * throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void publishSurvivorKeyVectors(int num) {
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ if (keyVectorQueues[i].isEmpty()) {
+ // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop
+ // is driven by that same count. Named rather than left to NoSuchElementException.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to "
+ + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num));
+ }
+ }
+ keyVectorsPublished = true;
+ // Queue k holds the survivors of key column k, and `keyColumnIndices[k]` is the batch slot that
+ // key column sits in, so the pairing is read off rather than re-derived from slot order. Only
+ // key slots are touched, since a non-key slot never holds anything but its persistent vector.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int k = 0; k < keyIndices.length; k++) {
+ spliceBatchColumns[keyIndices[k]] = keyVectorQueues[k].peekFirst();
Review Comment:
I kept the swap, and the pieces it needs are fewer than they were.
The copy would pay a value copy per surviving row at emit, on top of the one phase 1 already pays into the accumulator, and for a variable-length key that is a byte copy each. The swap is one array write per key column per batch. The `arraycopy` restore is gone, because the emit path now points every key slot at something on every batch rather than repairing the array afterwards. `closeAllocated` is the one override that replaces rather than extends, and it exists because the batch's key slots alias the queue heads. So what is left is the two arrays and the `resultBatch()` contract.
On the caller that holds a column across `nextBatch()`, `DataSourceReadBenchmark:235` does exactly that and is unaffected, because the plain reader never rewrites a slot. Only a caller that both asks for a storage filter and holds a column across batches would read a released vector, and `resultBatch()`'s javadoc now says the array may be rewritten per batch. If you would rather have the copy, it is contained enough for a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-6065406280
Thanks for the careful read, @viirya. Short answers below.
**Where it pays off**
- **The bloom has to exist first.** Right. #58635 adds a `RUNTIME_FILTER` hint that waives those heuristics, and it fits this feature well. A user who knows a join suits it can turn the conf on and hint that join. I tried it on top of this reader: on a broadcast join, which gets no bloom today, the hinted bloom let the fact scan skip 6 of its 10 row groups and return 100 of 100,000 rows.
- **Physical locality.** Your model is right, but the page that counts is the payload column's, not the key's. The bytes are in the wide columns, and those hold few rows per page. The benchmark's 2 kB column holds about 375 rows in a default page, so 0.1% scattered survivors still skip 65 MB of about 100 MB there. A narrow column holds up to 20,000 rows, as you say, but it is also cheap to read. The description says the page size decides the saving.
- **Delta and Iceberg.** The next two big steps we want to bring in are DSv2 support, which covers Iceberg, and per-partition bloom filters under SPJ. I'll also state the subclass exclusion in the conf doc. An explicit opt-in for subclasses is a good follow-up.
- **Object storage.** Agreed, that is the measurement that matters, and we plan to run one. The offset indexes are mostly not extra. A plain read loads the offset index of every projected column whenever a data filter is pushed, and an inner join's key gets an inferred `isnotnull` that is pushed. So the extra cost is one more round of vectored requests per row group. That is lost where the filter prunes little. Where it prunes a lot, slow IO is where we expect the larger gain.
- **The default.** A gating heuristic that gets this right for every workload is a hard problem, and that is why the conf is off by default. Until then the feature is for the workloads that clearly fit it, and the hint makes those easy to target.
**Survivor buffer memory.** Agreed. It is in the follow-up list as a `MemoryConsumer` whose `spill()` gives splicing up, and I'll move it up.
**Comments and the conf doc.** Agreed on both. I'll open a cleanup follow-up that keeps only the invariants and the non-obvious trade-offs, and your conf text is a good base for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5836432465
Thank you both. @dongjoon-hyun's third round found the one path in this change that could return a wrong answer, and @cloud-fan's blocking comment landed on the same survivor loop from the other side.
The change to look at first: `parquet.filter.columnindex.enabled=false` now turns this feature off entirely. Phase 0 honoured it, which was a half measure, because phase 2 reads part of a row group through the offset index whatever that conf says. A wrong index there pairs a row's key with another row's values, and the post-scan `Filter` cannot catch it, since the key it sees is the right one. The escape hatch is total now.
The rest of the round:
- One owner weighs the memory budget, after every surviving row, over the buffered key bytes and the row ranges together. A row group whose survivors fit in a single accumulator is weighed like any other, and the range half is far cheaper at the root: `ParquetReadState` coalesces lazily instead of materializing a list per column reader, so the `x leaves` factor is gone.
- The reader fails open on an evaluation error instead of the planner rejecting cast keys, so an `int` joined with `bigint` is pushed again.
- The phase-1 page store is closed at its own boundary, the key-slot pairing is read off the ordinals, the format is asked about the original expression rather than a canonicalized one, and the byte-metric walk reports an undercount instead of throwing, because `ignoreCorruptFiles` would turn a throw into a silently truncated file.
- Tests for what was untested: a file with no offset index at all, written with the low-level writer overloads you named, and an AQE arm that asserts the final plan's scan still carries the filter.
Three things are follow-ups in the description rather than in this PR: broadcasting the bloom bytes, letting a format declare its own metrics, and coarsening the survivor ranges instead of giving the filter up. One is a reasoned no, in its own thread: the offset-index footer preflight, which round 2 of this PR had already deleted for two reasons my new attempt had again.
The description is restructured, with the phases and the six ways the reader stops short up front. The suite is at 99 tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5890470420
@dongjoon-hyun @cloud-fan thank you both. Round 4 is answered: every comment is either fixed in the code or answered with a reason in its thread, and the description is up to date with all of it.
**Three of them changed the design rather than the code around it.** `parquet.filter.columnindex.enabled=false` no longer turns the feature off, it leaves the filter with the row groups it empties. A missing offset index is decided from the footer per row group, before anything is buffered, with parquet's exception kept as the backstop. And the missing-key rewrite is gone, so the row phase 1 evaluates takes the vectors `initBatch` built, which leaves one rule for what a missing column reads as.
**There is a large refactor in this push, and I would like your read on it.** The read path is a subclass now, `LateMaterializationParquetRecordReader`, so `VectorizedParquetRecordReader` keeps the plain path and nothing else. Three things that are not about this feature stand on their own as well: a per-type `ValueCopier`, a `ParquetBlockAccounting` for the footer arithmetic, and the metric keys and labels on `StorageFilterMetrics`'s companion.
My reason is reviewability. The single-file version was hard to read by hand, and the base class's diff against master is now five things with no plain-path logic among them, which can be checked on its own. It does change code you have already reviewed, so if you would rather have the shape you reviewed, say so and I will revert it, or move it into a PR of its own that lands before or after this one. The correctness fixes from this round are in separate commits, so a revert of the refactor does not take any of them with it.
**The scattered-survivor measurement** is in the description. One long key and twenty long payload columns, 256-row pages, one survivor every other row: phase 2 skips no page at all and the read takes 5.6x the plain one, falling to 1.8x at runs of sixteen rows. So a gate is needed. It is a follow-up rather than part of this PR, because giving up also re-reads the key columns and hands the post-scan `Filter` every row the filter would have dropped, and those costs grow with what was excluded rather than with what was kept, so the direction flips as the filter gets more selective.
**Two things came out of my own review rounds after yours.** The flag that says the page index cannot be used carried two meanings, and for the one a failed partial read latches, dropping `getRowRanges` for the rest of the file dropped the pushed data filter's own page pruning too. Measured at 215 rows where a plain read of the same query hands back 160, so the feature read more than not having it. There are two flags now, and a test on a hand-damaged file pins it, which is the first test to reach that retry at all. The other is the survivor budget, which charged a variable-length key 9 bytes per row where the vectors allocate 17, so a null-heavy string key could hold twice the cap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4148902119
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, TaskContext}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. None counts what it cost. A row group
+ * whose filter was given up reads its key columns twice and counts nothing, so the counters can
+ * show a saving on a scan that read more than it would have with the feature off.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums those skipped row groups' rows, counting per block the rows
+ * that survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned. A row group that gave
+ * splicing up reads its key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+
+ /** These counters keyed for the scan, which carries them without naming any of them. */
+ def toMap: Map[String, SQLMetric] = Map(
+ StorageFilterMetrics.ROW_GROUPS_SKIPPED -> rowGroupsSkipped,
+ StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP -> rowsExcludedByRowGroup,
+ StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP -> rowsExcludedWithinRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP -> bytesAvoidedByRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING -> bytesAvoidedByPageFiltering)
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /** A fresh set of counters, with the labels the SQL UI shows. */
+ def create(sparkContext: SparkContext): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped =
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ rowsExcludedByRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ rowsExcludedWithinRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ bytesAvoidedByRowGroup = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ bytesAvoidedByPageFiltering = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made: the map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer
+ // construction to first use on the executor.
+ @transient private lazy val predicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ require(boundExpression != null,
+ "a storage filter must only be evaluated on an executor, and this copy has released the " +
+ "expression it would build its predicate from")
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay.
+ */
+ def preparedPredicate: BasePredicate = predicate
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queues where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader evaluates
+ * the predicate without the conjuncts that precede it in the plan, so one that throws on a row an
+ * earlier conjunct would have rejected throws where a plain scan does not. That is handled where
+ * it arises. The reader gives the filter up for the row group and reads it plainly, and the
+ * post-scan `Filter` then evaluates every conjunct in its own order.
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows, as it is, an error the reader met while applying a storage filter that it must not
+ * absorb by giving the filter up. Two kinds are not the reader's to absorb, and both are asked
+ * the way the rest of Spark asks them:
+ * - a pending task kill, since the error may be the kill's interrupt surfacing through a UDF
+ * that wrapped it. `PythonRunner` reads a kill off the task context in the same way.
+ * - a fatal error in the cause chain, found by `Executor.isFatalError` as deep as the executor
+ * itself looks, which is `spark.executor.killOnFatalError.depth`.
+ *
+ * As it is, because a `TaskKilledException` of the reader's own would be read as a corrupt file
+ * under `ignoreCorruptFiles`.
+ */
+ def rethrowIfMustPropagate(e: Throwable): Unit = {
+ val context = TaskContext.get()
+ val killed = context != null && context.isInterrupted()
+ if (killed || Executor.isFatalError(e, fatalErrorDepth)) throw e
Review Comment:
**Rethrowing "as it is" does not get past `FileScanRDD` under `ignoreCorruptFiles` when the original is a `RuntimeException` or an `IOException`, and the reader's own survivor buffer raises exactly such a wrapper.** `WritableColumnVector.reserve` (L136-139, L159) turns an `OutOfMemoryError` into `new RuntimeException("Cannot reserve additional contiguous bytes ...", oom)`. Phase 1 reaches it inside the per-row `try` (`LateMaterializationParquetRecordReader.java:742-778`), from the copy at L873 (`putByteArray` -> `appendBytes` -> `reserve`) and from the round-6 up-front `accumulator.arrayData().reserve(valueBytes)` at L847. Then:
- `isFatalError` finds the OOM in the cause chain, so L775 rethrows the `RuntimeException`.
- With `spark.sql.files.ignoreCorruptFiles=true`, `FileScanRDD` (L284-289) matches it with `DataSourceUtils.shouldIgnoreCorruptFileException` (`RuntimeException | IOException | InternalError`) and logs "Skipped the rest of the content in the corrupted file". The task succeeds without the rest of the split. A plain read never allocates this buffer and returns those rows, and the executor never sees the fatal error.
- At 058b3cafae4 the same exception was absorbed: `abandonSplicing()` released the buffer and the row group was read plainly. Its comment said so ("Copying a surviving key lands here too, which a survivor vector that cannot grow does, and reading the row group with nothing buffered is the answer to that as well"). The comment at L770-774 still claims that answer.
A pending kill has the same gap. With a kill pending and, say, an ANSI `CAST` key that throws `SparkNumberFormatException`, the rethrown exception is logged as a corrupt file. `nextIterator()` then opens the next file without a kill check (only `hasNext` checks, `FileScanRDD.scala:149`), and every file whose phase 1 meets such a row repeats that. The scaladoc's reason, that a `TaskKilledException` is a `RuntimeException`, holds for the original too. Only a checked exception such as the Hive UDF's `SparkException` escapes, and that is the one shape the new test covers, with `ignoreCorruptFiles` off.
Could the rethrow use a form that `shouldIgnoreCorruptFileException` rejects, e.g. wrap it in a checked `SparkException`? The executor still finds a fatal cause within `spark.executor.killOnFatalError.depth`, and still reports a killed task as killed. For the survivor buffer itself, round 5's answer, releasing it and reading the row group plainly, may still be the better one, since that allocation is optional.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1947,6 +1947,57 @@ object SQLConf {
.checkValue(threshold => threshold >= 0, "The threshold must not be negative.")
.createWithDefault(10)
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED =
+ buildConf("spark.sql.parquet.storageFilterPushdown.enabled")
+ .doc("If true, the vectorized Parquet reader may apply a runtime storage filter, such as a " +
+ "bloom filter from join runtime filtering, while it reads. It reads the columns the " +
+ "filter needs first, evaluates the filter per row, and then reads the remaining columns " +
+ "only for the rows that survived. This is a planning-time decision. " +
+ "A filter that is attached also stays in the post-scan filter, the way a pushed data " +
+ "filter does, so the reader is free to stop applying it wherever doing so would cost " +
+ "more than it saves, and the answer does not change. A row group where it stops reads " +
+ "its key columns twice, which is slower than not pushing the filter. " +
+ "Under spark.sql.files.ignoreCorruptFiles the answer can change, because this reader " +
+ "reads different pages in a different order than a plain read. Which rows survive a " +
+ "corrupt page can then differ from a plain read, in either direction. " +
+ "Returning only the surviving rows of a row group relies on the Parquet page index, so a " +
Review Comment:
**The round-6 rewrite dropped what made this caveat specific to this feature.** At 058b3cafae4 the doc said: "Narrowing a row group to part of its rows trusts the Parquet page index, which a read with no pushed data filter never consults, so on a file whose page index is wrong this can pair a row's key with another row's values. Turning this off avoids that." That was the answer to the earlier OffsetIndex thread (https://github.com/apache/spark/pull/58895#discussion_r4108708237). The paragraph now ends with "The page index and that conf also apply to the page-level filtering of pushed data filters, with or without this feature". Right after the hazard, that reads as if the exposure predates this feature.
For a bloom-only query it does not:
- With the conf off, the plain read goes from `readNextFilteredRowGroup` to `readNextRowGroup` when no filter is pushed (parquet 1.18.1 `ParquetFileReader` L1403-1404), so it never reads an offset index.
- With the conf on, phase 2's `readFilteredRowGroup(blockIdx, finalRanges)` (`LateMaterializationParquetRecordReader.java:550`) takes the pages and their first row indexes from the offset index. A wrong index then pairs a correct spliced key with another row's values, and the post-scan Filter only re-checks the key.
The description has the same gap: "The one exception is under `spark.sql.files.ignoreCorruptFiles`" in the user-facing section, and "`ignoreCorruptFiles` is also the one setting under which the answer can differ". Could both say again that bloom-only reads start trusting the offset index with this feature, and that turning it off avoids that?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
+ * read for the rest of this split. The pushed data filter's own page pruning still applies, since
+ * parquet builds those ranges and hands back whole any block it has no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a read that throws in between leaves them reachable for {@link #close()}.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key column.
+ List<Type> requestedFields = requestedSchema.getFields();
+ boolean anyNonKeyPresent = false;
+ for (int i = 0; i < requestedFields.size() && !anyNonKeyPresent; i++) {
+ anyNonKeyPresent = !keyTopLevelNames.contains(requestedFields.get(i).getName())
+ && !missingColumns.contains(parquetColumn.children().apply(i));
+ }
+ if (!anyNonKeyPresent) return;
+ nonKeyColumns = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
Review Comment:
**The reader's own loops never check for a task kill, so a kill can wait for the rest of the task's partition.** A plain read comes back to `FileScanRDD.hasNext`, which calls `killTaskIfInterrupted()` (`FileScanRDD.scala:149`), after every batch. Here one `hasNext` can run all of these:
- this loop over every row group the filter empties (the `continue`s at L496 and L538-539);
- phase 1's chunk loop (L723-781), which decodes and evaluates a whole row group;
- when a whole file is emptied, `FileScanRDD.nextIterator()` (L302-305), which goes on to the next file, still without a kill check.
So a cooperative kill (e.g. a job group cancelled without `interruptOnCancel`), or an interrupting one on a stream that ignores interrupts such as the local FS, is seen only when a row group with survivors emits a batch. The answer is not affected, since the executor checks the kill when the task ends. But the task keeps reading and evaluating key columns after it was killed.
Related: with no pushed data filter, phase 2's narrowed read is the first to read the offset indexes. Parquet's `ColumnIndexStoreImpl` turns any `IOException` there, including an interrupt's `InterruptedIOException` or `ClosedByInterruptException`, into `MissingOffsetIndexException`. The catch at L551 then latches `partialReadUnavailable`, logs the offset-index WARN and re-reads the row group, without `rethrowIfMustPropagate`.
Could this loop check for a kill per row group, and phase 1 per chunk? `killTaskIfInterrupted()` throws a `TaskKilledException`, which the `ignoreCorruptFiles` catch in `FileScanRDD` would swallow too, so it needs the same care as `rethrowIfMustPropagate`.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2757 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkException, TaskContext}
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading the
+ // rest of a file without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender("partial-read warning")
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val warned = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering"))
+ (result.get, warned)
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
Review Comment:
Minor: `result.size` is only in the message here, so this passes even if every row came back twice. The same goes for every planner-level result check on a storage-filter scan: L599-601, L631-635, L2039-2044, L2226-2228, L2246, L2309, L2337-2341, L2365-2369, L2392-2393, L2424-2434, and the schema-evolution tests. They all compare Sets, so row multiplicity is checked only by the reader- and format-level tests. Duplication added above the reader, or in a branch only these tests reach, would pass. Those branches are a missing key or non-key column, `_metadata.row_index`, a struct non-key column and column-index filtering off. The post-scan Filter does not remove duplicates either. Could these also compare sizes, or sorted sequences?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2757 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkException, TaskContext}
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading the
+ // rest of a file without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender("partial-read warning")
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val warned = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering"))
+ (result.get, warned)
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString),
+ useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(195L))
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
Review Comment:
**This test can no longer fail.** Its premise ("reached the filter it had just released and failed with an NPE") is gone, because round 6 made `storageFilter` final. Suppose `VectorizedParquetRecordReader.initialize(InputSplit, TaskAttemptContext)` (L191-194) got master's second `initializeInternal()` back. This all-keys projection would then decline again at `LateMaterializationParquetRecordReader.java:437` and read all 20 rows, so the test passes whether initialization runs once or twice.
Round 6 also dropped the `keyFixedBytesPerRow = 0;` before the `+=` at L459. So a second pass on a file that is not declined would double the per-row charge (9 -> 18 bytes for a long key, 17 -> 34 for a string) and give splicing up at half the cap, and no test would notice.
Could the test use a projection with a non-key column, and observe something a second pass changes, e.g. a cap that one charge stays under and a doubled one crosses? Resetting the field, or summing into a local, would also make a second pass harmless again. The description's "which used to fail with an NPE" needs the same update.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, TaskContext}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. None counts what it cost. A row group
+ * whose filter was given up reads its key columns twice and counts nothing, so the counters can
+ * show a saving on a scan that read more than it would have with the feature off.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums those skipped row groups' rows, counting per block the rows
+ * that survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned. A row group that gave
+ * splicing up reads its key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+
+ /** These counters keyed for the scan, which carries them without naming any of them. */
+ def toMap: Map[String, SQLMetric] = Map(
+ StorageFilterMetrics.ROW_GROUPS_SKIPPED -> rowGroupsSkipped,
+ StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP -> rowsExcludedByRowGroup,
+ StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP -> rowsExcludedWithinRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP -> bytesAvoidedByRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING -> bytesAvoidedByPageFiltering)
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /** A fresh set of counters, with the labels the SQL UI shows. */
+ def create(sparkContext: SparkContext): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped =
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ rowsExcludedByRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ rowsExcludedWithinRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ bytesAvoidedByRowGroup = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ bytesAvoidedByPageFiltering = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made: the map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer
+ // construction to first use on the executor.
+ @transient private lazy val predicate: BasePredicate = {
Review Comment:
Nit: since `test(keyRow)` was removed (2b138116958), `preparedPredicate` (L153) is the only reader of this `private lazy val`. Declaring it as `@transient lazy val preparedPredicate`, with that scaladoc, would drop the second name. This is the one place with the "this lazy val is the only reader of the field" invariant. The Java callers keep calling `preparedPredicate()`.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -151,6 +151,66 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
}
}
+ /**
+ * The conjuncts of `afterScanFilters` the file format can evaluate at the storage layer for late
+ * materialization, to prune the IO of the columns the filter does not reference.
+ *
+ * They stay in the post-scan `Filter` as well, the way a pushed data filter does. The reader is
+ * offered them, not obliged to honor them, so the plan keeps the exact check. What it costs is
+ * evaluating the conjunct a second time for the rows the scan emits. What it buys is a reader
+ * free to give up on a file it cannot read in part, or on a stretch of rows whose survivors cost
+ * too much to keep track of, without the answer depending on it.
+ *
+ * Two of the conditions are per scan, and failing either offers nothing:
+ * - [[FileFormat.supportsStorageFilterPushdown]] holds. That is where a format reads the conf
+ * that enables this, so a format's own conf never decides for another format, and asking it
+ * first keeps everything below off the path of a scan that will not use it.
+ * - [[FileFormat.supportBatch]] holds for the schema the reader will see,
+ * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives
+ * its own vectorized-read decision from. Late materialization needs a batch read, so this asks
+ * about batch support rather than naming a format.
+ *
+ * The rest are per conjunct:
+ * - It is deterministic. The reader drops the rows the conjunct rejects and the post-scan
+ * `Filter` evaluates it again on the rows it keeps, so the two evaluations have to agree.
+ * - It references at least one column, and every column it references is a projected data
+ * column. A reference to something the scan does not read cannot be evaluated by the reader.
+ * - [[FileFormat.supportsStorageFilter]] accepts it. That is where the expression shapes and
+ * column types a reader can evaluate live, so this method names neither a format nor a type.
+ *
+ * One last condition is on the set that survives, that at least one projected column is left for
+ * the reader to prune. A scan that projects nothing but the filter's own storage-key columns
+ * reads the same columns for the same rows either way, since the reader has to read a key column
+ * to evaluate the filter on it, so offering it could only add the cost of evaluating the
+ * predicate outside the generated code. Nothing is offered in that case.
+ */
+ private def storageFiltersFor(
+ afterScanFilters: ExpressionSet,
+ fsRelation: HadoopFsRelation,
+ readDataColumns: Seq[Attribute],
+ outputDataSchema: StructType): Seq[Expression] = {
+ val sparkSession = fsRelation.sparkSession
+ if (!fsRelation.fileFormat.supportsStorageFilterPushdown(sparkSession)) return Nil
+ val resultSchema = StructType(fsRelation.partitionSchema.fields ++ outputDataSchema.fields)
+ if (!fsRelation.fileFormat.supportBatch(sparkSession, resultSchema)) return Nil
+
+ val dataAttrs = AttributeSet(readDataColumns)
+ // Over `toSeq` rather than the set, because `ExpressionSet.filter` hands the predicate
+ // `e.canonicalized`, which drops attribute names and metadata, and a format deciding by either
+ // would answer about an expression it will never be given. Normalized to the relation's own
+ // column names, as the pushed data filters are, because the analyzer spells a reference the way
+ // the query did, while the reader sees the names of the schema it reads.
+ val offered = DataSourceStrategy.normalizeExprs(afterScanFilters.toSeq, readDataColumns)
Review Comment:
Minor: every conjunct this can offer is already in `normalizedFilters` (L225-226), under the same names:
- an offered conjunct is deterministic and references only `readDataColumns`;
- `afterScanFilters` removes only partition-key filters;
- `l.output` and `readDataColumns` spell data columns the same way, the data schema's.
Could `storageFiltersFor` take `ExpressionSet(normalizedFilters).toSeq`, to keep the dedup, instead of normalizing a second time against `readDataColumns` and re-checking `deterministic`? The comment at L198-202 could go too. The trade-off is that the name agreement with `create` would then rest on `l.output`'s spelling, which holds for every in-tree relation.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
+ * read for the rest of this split. The pushed data filter's own page pruning still applies, since
+ * parquet builds those ranges and hands back whole any block it has no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a read that throws in between leaves them reachable for {@link #close()}.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key column.
+ List<Type> requestedFields = requestedSchema.getFields();
+ boolean anyNonKeyPresent = false;
+ for (int i = 0; i < requestedFields.size() && !anyNonKeyPresent; i++) {
+ anyNonKeyPresent = !keyTopLevelNames.contains(requestedFields.get(i).getName())
+ && !missingColumns.contains(parquetColumn.children().apply(i));
+ }
+ if (!anyNonKeyPresent) return;
+ nonKeyColumns = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped, with every projected byte of the rows the pushed data filter kept as
+ * avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Exception e) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. Giving the row group up hands it to phase 2 whole, which reads
+ // the same pages in the same batches as a plain scan. So the page fails at the same batch,
+ // after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain read.
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) splicedBytes += appendSurvivorRowToAccumulators(r);
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together
+ // here.
+ long rangeBytes = survivorRangeCount * ROW_RANGE_BYTES;
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ } catch (Exception e) {
+ // Fail open on any exception. The predicate runs here without the conjuncts before it in
+ // the plan, so it can throw on a row a plain scan never evaluates it on (see
+ // `ParquetStorageFilter.isSupportedStorageFilter`). Giving the filter up returns every
+ // row the pushed filter kept, and the post-scan Filter decides them in its own order.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK error
+ // (`timestamp_seconds` of a decimal), and a Hive UDF throws a checked SparkException.
+ // Rethrowing would fail a query a plain read answers, or drop the rest of the file
+ // silently under `ignoreCorruptFiles`. Only a task kill or a fatal error is rethrown,
+ // first. A failed copy of a surviving key lands here too, and gets the same answer.
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, ROW_GROUP_GIVEN_UP_ON_EVALUATION);
+ return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, and returns what that costs the
+ * budget up front. A variable-length key's byte child is sized to what the set queued before it
+ * held, so it does not grow from the default one doubling at a time for every capacity's worth
+ * of survivors. That memory is taken before any value arrives, so it is charged now, as if those
+ * values had arrived, and the values written into it use that charge up before they add to it.
+ */
+ private long allocateKeyAccumulators() {
+ WritableColumnVector[] previous = survivorBatches.peekLast();
+ // Assigned before the loop, so an allocation failure part way through leaves the vectors
+ // allocated so far reachable for `abandonSplicing`.
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ long charged = 0L;
+ for (int i = 0; i < keyColumns.length; i++) {
+ WritableColumnVector accumulator = newKeyVector(i);
+ currentKeyAccumulators[i] = accumulator;
+ prepaidValueBytes[i] = 0L;
+ if (keyColumns[i].variableLength() && previous != null) {
+ int valueBytes = previous[i].arrayData().getElementsAppended();
+ accumulator.arrayData().reserve(valueBytes);
+ prepaidValueBytes[i] = VARIABLE_LENGTH_BYTES_FACTOR * valueBytes;
+ charged += prepaidValueBytes[i];
+ }
+ }
+ return charged;
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, allocating them first if
+ * there are none, and hands them to the queue once they are full. Returns the bytes the row
+ * added, which the caller weighs against its budget.
+ */
+ private long appendSurvivorRowToAccumulators(int srcRow) {
+ long charged = currentKeyAccumulators == null ? allocateKeyAccumulators() : 0L;
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final KeyColumn[] keys = keyColumns;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ KeyColumn key = keys[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ key.copier().copy(dst, dstRow, src, srcRow);
+ // Measured on the destination, because a dictionary-encoded source has no length of its
+ // own, its values being read through the dictionary.
+ if (key.variableLength()) {
+ long valueCharge = VARIABLE_LENGTH_BYTES_FACTOR * dst.getArrayLength(dstRow);
+ long prepaid = Math.min(prepaidValueBytes[i], valueCharge);
+ prepaidValueBytes[i] -= prepaid;
+ charged += valueCharge - prepaid;
+ }
+ }
+ }
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) pushAccumulators();
+ return keyFixedBytesPerRow + charged;
+ }
+
+ /** Hands the accumulators to the queue the emit path takes its batches from. */
+ private void pushAccumulators() {
+ survivorBatches.addLast(currentKeyAccumulators);
+ currentKeyAccumulators = null;
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /** Releases the pages phase 2 read for the row group just emitted, if any. */
+ private void closeDataPages() {
+ if (dataPages != null) {
+ dataPages.close();
+ dataPages = null;
+ }
+ }
+
+ /** An error applying the filter to a row gave one row group's filter up. */
+ private static final String ROW_GROUP_GIVEN_UP_ON_EVALUATION =
+ "Reading a row group of {} without the storage filter, because applying it to a row raised "
+ + "an error. The filter is still applied above the scan, so the answer is unchanged, and "
+ + "the remaining row groups are filtered as usual";
+
+ /** An error decoding a key page gave one row group's filter up. */
+ private static final String ROW_GROUP_GIVEN_UP_ON_DECODING =
+ "Reading a row group of {} without the storage filter, because decoding its key columns "
+ + "raised an error. The row group is read the way a plain scan reads it, which raises "
+ + "the same error if the data is corrupt, and the remaining row groups are filtered as "
+ + "usual";
+
+ /** An error applying the filter to a file's constant key values gave the whole file up. */
+ private static final String FILE_GIVEN_UP_ON_EVALUATION =
+ "Reading {} without the storage filter, because applying it to the values its missing key "
+ + "columns read as raised an error. The filter is still applied above the scan, so the "
+ + "answer is unchanged";
+
+ /**
+ * Reports the first time this reader gives the filter up on an error, in one of the messages
+ * above. Once per reader, which is once per split, because a file whose values do that tends to
+ * do it again.
+ */
+ private void logFilterGivenUpOnError(Exception e, String message) {
Review Comment:
Minor: all three give-up catches (L331-335, L730-738, L764-778) have to call `rethrowIfMustPropagate` and then this method, in that order. Only the per-row site has a kill/fatal test (`ParquetStorageFilterSuite.scala:526-557`), so deleting the rethrow at L332 or L735 leaves the suite green. R6-2 was exactly a catch that gave up without it. Could one helper do both, rethrow first and then the once-per-reader WARN, so that a new catch cannot take one without the other? It would also be the one place to change for the `ignoreCorruptFiles` comment on `ParquetStorageFilter.scala:262`.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
+ * read for the rest of this split. The pushed data filter's own page pruning still applies, since
+ * parquet builds those ranges and hands back whole any block it has no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a read that throws in between leaves them reachable for {@link #close()}.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key column.
+ List<Type> requestedFields = requestedSchema.getFields();
+ boolean anyNonKeyPresent = false;
+ for (int i = 0; i < requestedFields.size() && !anyNonKeyPresent; i++) {
+ anyNonKeyPresent = !keyTopLevelNames.contains(requestedFields.get(i).getName())
+ && !missingColumns.contains(parquetColumn.children().apply(i));
+ }
+ if (!anyNonKeyPresent) return;
+ nonKeyColumns = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped, with every projected byte of the rows the pushed data filter kept as
+ * avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Exception e) {
Review Comment:
**`catch (Exception e)` misses a corrupt key page that the codec reports as `java.lang.InternalError`, so the key-page claim does not hold for it.** hadoop-lzo's `LzoDecompressor` throws `InternalError("lzo1x_decompress_safe returned: N")` for corrupt input, and `lzo` is an accepted Parquet codec. `DataSourceUtils.shouldIgnoreCorruptFileException` treats `InternalError` as corruption since SPARK-52482. Take `ignoreCorruptFiles=true` and such a key page at or after the second batch of row group G:
- a plain read emits G's earlier batches and fails at the corrupt one;
- here phase 1 fails before any batch of G is emitted, so those rows are lost.
Widening the catch alone would not help, since `rethrowIfMustPropagate` treats an `InternalError` as fatal and rethrows it. The conf doc's "in either direction" already allows this, so it is not an undisclosed change of the answer. But three places overstate it: the description ("A corrupt key page is the exception, since giving the row group up meets it at the same batch as a plain read"), the comment at L731-734, and the test comment at `ParquetStorageFilterSuite.scala:1103-1108`. Could they be narrowed to decode errors raised as exceptions?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
Review Comment:
Nit: "If so, no part of a row group is read for the rest of this split" contradicts the next sentence and the code. After the latch, phases 1 and 2 still read `pushedFilterRanges` (L516-517, L527, L565), which parquet narrows for a block whose index is intact. The damaged-index test pins that (`ParquetStorageFilterSuite.scala:1194-1197`). The old wording, "Only the narrowing goes", said it right. The test comment at `ParquetStorageFilterSuite.scala:1017-1019` has the same sentence, and still says "file".
Similarly, `recordFileSkipped`'s javadoc (L608-610) says "Every row group counts as skipped". But L620-621 skip a block the pushed data filter empties without counting it, which follows the main loop's "Not this filter's skip" (L494-496) rather than the javadoc.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
+ * read for the rest of this split. The pushed data filter's own page pruning still applies, since
+ * parquet builds those ranges and hands back whole any block it has no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
Review Comment:
Nit: "so a read that throws in between leaves them reachable for `close()`" was round 5's reason, when the key slots were published before the base read the batch. Since round 6 they are released before `super.nextBatch()` (L355-357) and published after it (L360-374), so no read can throw while this field is set. What it is needed for now is a consumer that stops between batches, such as a LIMIT, the end of the task or a downstream error. Then `close()` frees the survivors that already left the queue (L953-954). Could the javadoc say that?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, TaskContext}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. None counts what it cost. A row group
+ * whose filter was given up reads its key columns twice and counts nothing, so the counters can
+ * show a saving on a scan that read more than it would have with the feature off.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums those skipped row groups' rows, counting per block the rows
+ * that survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned. A row group that gave
+ * splicing up reads its key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+
+ /** These counters keyed for the scan, which carries them without naming any of them. */
+ def toMap: Map[String, SQLMetric] = Map(
+ StorageFilterMetrics.ROW_GROUPS_SKIPPED -> rowGroupsSkipped,
+ StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP -> rowsExcludedByRowGroup,
+ StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP -> rowsExcludedWithinRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP -> bytesAvoidedByRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING -> bytesAvoidedByPageFiltering)
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /** A fresh set of counters, with the labels the SQL UI shows. */
+ def create(sparkContext: SparkContext): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped =
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ rowsExcludedByRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ rowsExcludedWithinRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ bytesAvoidedByRowGroup = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ bytesAvoidedByPageFiltering = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made: the map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer
+ // construction to first use on the executor.
+ @transient private lazy val predicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ require(boundExpression != null,
+ "a storage filter must only be evaluated on an executor, and this copy has released the " +
+ "expression it would build its predicate from")
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay.
+ */
+ def preparedPredicate: BasePredicate = predicate
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queues where the row group was spliced, from the overwritten vector where it
Review Comment:
Nit: round 6 replaced the per-key-column queues with the single `survivorBatches` queue of per-batch arrays (`LateMaterializationParquetRecordReader.java:203`), and the reader's comments now say "survivor queue". A few places still use the per-column plural:
- this line;
- `ParquetStorageFilterSuite.scala:1211` ("splice queues");
- L2572 ("The survivor queues");
- the test name at L2518 ("emit maps queues to the right batch slots"), whose own comment at L2519 now says "Each key column's survivors go back to that column's batch slot".
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
+ * read for the rest of this split. The pushed data filter's own page pruning still applies, since
+ * parquet builds those ranges and hands back whole any block it has no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a read that throws in between leaves them reachable for {@link #close()}.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key column.
+ List<Type> requestedFields = requestedSchema.getFields();
+ boolean anyNonKeyPresent = false;
+ for (int i = 0; i < requestedFields.size() && !anyNonKeyPresent; i++) {
Review Comment:
**Minor: this decline misses a struct whose requested leaves are all missing from the file.** It tests each top-level `ParquetColumn` against `missingColumns`. But `checkColumn` (`VectorizedParquetRecordReader.java:440-461`) records only the missing leaves of a struct the file has, never the struct itself.
By default the clipper appends a leaf the file has (`ParquetReadSupport.scala:511-516`), so phase 2 still has pages to prune. With `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` it does not. Take a table `(k BIGINT, s STRUCT<a INT, b INT>)` where `s.b` was added later, and `SELECT f.s.b FROM fact f JOIN dim d ON f.k = d.k`. Each older file:
- is not declined, since `s` is not in `missingColumns`;
- has phase 2 request only `s.b`, which reads nothing;
- still pays phase 1's per-row evaluation and buffering, and without an offset index on `k` reads `k` twice.
That is the cost R6-5 set out to remove. Could the decline instead ask whether any of `nonKeyColumns` exists in the file schema, e.g. with `fileSchema.containsPath(c.getPath())`? That covers the top-level case, the row-index column and this one.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,980 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does the rest of a split once a partial read has failed.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether {@link #logFilterGivenUpOnError} has already reported. */
+ private boolean loggedFilterGivenUpOnError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a partial read has failed for want of an offset index. If so, no part of a row group is
+ * read for the rest of this split. The pushed data filter's own page pruning still applies, since
+ * parquet builds those ranges and hands back whole any block it has no index for.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a read that throws in between leaves them reachable for {@link #close()}.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Named here rather than left to a null batch
+ // slot and an NPE in the consumer.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key column.
+ List<Type> requestedFields = requestedSchema.getFields();
+ boolean anyNonKeyPresent = false;
+ for (int i = 0; i < requestedFields.size() && !anyNonKeyPresent; i++) {
+ anyNonKeyPresent = !keyTopLevelNames.contains(requestedFields.get(i).getName())
+ && !missingColumns.contains(parquetColumn.children().apply(i));
+ }
+ if (!anyNonKeyPresent) return;
+ nonKeyColumns = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline is the non-key bytes a plain read transfers for every row the
+ // pushed filter kept. That is all this feature can avoid. It is not walked for a row group
+ // whose filter was given up, since nothing is reported for it.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the block is skipped whole and the whole non-key
+ // baseline is avoided. Phase 1's key read is not part of the baseline, so nothing is
+ // subtracted for it.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ // Phase 2 reads the non-key columns over the survivors while splicing, and the whole
+ // projection otherwise.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // the block's store. The row group is then read again over `pushedFilterRanges`, which that
+ // empty store has already widened to the whole block, so the retry cannot throw.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The message only, since parquet has already logged the failed index read with its stack
+ // trace.
+ LOG.warn("Reading the rest of {} without page-level storage filtering, because parquet "
+ + "could not read the offset index of a column the read needs ({}). Row groups the "
+ + "filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // Measured over the columns phase 2 read, which after giving splicing up are the key
+ // columns as well, a second read of them that the baseline does not count. A spliced row
+ // group that pruned nothing read exactly the baseline, which is not walked a second time.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped, with every projected byte of the rows the pushed data filter kept as
+ * avoided, which is what the counters mean for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw new IllegalStateException(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Exception e) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. Giving the row group up hands it to phase 2 whole, which reads
+ // the same pages in the same batches as a plain scan. So the page fails at the same batch,
+ // after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain read.
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) splicedBytes += appendSurvivorRowToAccumulators(r);
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together
+ // here.
+ long rangeBytes = survivorRangeCount * ROW_RANGE_BYTES;
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ } catch (Exception e) {
+ // Fail open on any exception. The predicate runs here without the conjuncts before it in
+ // the plan, so it can throw on a row a plain scan never evaluates it on (see
+ // `ParquetStorageFilter.isSupportedStorageFilter`). Giving the filter up returns every
+ // row the pushed filter kept, and the post-scan Filter decides them in its own order.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK error
+ // (`timestamp_seconds` of a decimal), and a Hive UDF throws a checked SparkException.
+ // Rethrowing would fail a query a plain read answers, or drop the rest of the file
+ // silently under `ignoreCorruptFiles`. Only a task kill or a fatal error is rethrown,
+ // first. A failed copy of a surviving key lands here too, and gets the same answer.
+ ParquetStorageFilter.rethrowIfMustPropagate(e);
+ logFilterGivenUpOnError(e, ROW_GROUP_GIVEN_UP_ON_EVALUATION);
+ return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, and returns what that costs the
+ * budget up front. A variable-length key's byte child is sized to what the set queued before it
+ * held, so it does not grow from the default one doubling at a time for every capacity's worth
+ * of survivors. That memory is taken before any value arrives, so it is charged now, as if those
+ * values had arrived, and the values written into it use that charge up before they add to it.
+ */
+ private long allocateKeyAccumulators() {
+ WritableColumnVector[] previous = survivorBatches.peekLast();
+ // Assigned before the loop, so an allocation failure part way through leaves the vectors
+ // allocated so far reachable for `abandonSplicing`.
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ long charged = 0L;
+ for (int i = 0; i < keyColumns.length; i++) {
+ WritableColumnVector accumulator = newKeyVector(i);
+ currentKeyAccumulators[i] = accumulator;
+ prepaidValueBytes[i] = 0L;
+ if (keyColumns[i].variableLength() && previous != null) {
+ int valueBytes = previous[i].arrayData().getElementsAppended();
Review Comment:
**Minor: nothing checks the key values of a pre-reserved accumulator set.** This path runs only for a variable-length key with more than `capacity` survivors in one row group. The only test that reaches it, "an accumulator sized up front is charged ..." (`ParquetStorageFilterSuite.scala:871`), reads through `readCounting`, which returns only row counts and bytes. The key-type matrix runs at capacity 4096 over files of about 100 rows, and every small-capacity test uses a long key. So a regression that corrupted the reserved byte child would return wrong keys for survivors past `capacity`, with all tests green. Could that test's 64MB arm also compare `(k, v)` pairs, e.g. through `readAllWith(..., capacity = 4)`?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2757 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkException, TaskContext}
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading the
+ // rest of a file without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender("partial-read warning")
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val warned = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering"))
+ (result.get, warned)
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString),
+ useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(195L))
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs the reader's own initialization, and it
+ // used to run that a second time on top. A reader that declined the filter on the first pass,
+ // here because every projected column is a key column, then reached the filter it had just
+ // released and failed with an NPE instead of reading plainly.
+ withTempDir { dir =>
+ val keys = (1L to 20L)
+ val path = writeKeyParquetFile(dir, keys)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(15L))),
+ requested)
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ Utils.tryWithResource(new LateMaterializationParquetRecordReader(false, 4096, filter)) {
+ reader =>
+ reader.initialize(split, context)
+ reader.initBatch(new StructType(), null)
+ val read = mutable.ArrayBuffer[Long]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ assert(read == keys, s"every row must come back unfiltered; got ${read.size}")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(0L))),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = ThrowsCheckedOn(BoundReference(0, LongType, nullable = false), bad = 7L)
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises a checked SparkException, so the kill is
+ // read off the task context rather than off the error. A fatal error a UDF wrapped is found in
+ // the cause chain, which is where the executor looks for one. Both are rethrown as they are.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def readWith(key: Expression): Unit = {
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ readAll(path, filter)._2.close()
+ }
+
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.setTaskContext(context)
+ val killed = try {
+ intercept[SparkException](readWith(ThrowsCheckedOn(bound, bad = 7L)))
+ } finally {
+ TaskContext.unset()
+ }
+ assert(killed.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task; got $killed")
+
+ val fatal = intercept[SparkException](
+ readWith(ThrowsCheckedOn(bound, bad = 7L, fatalCause = true)))
+ assert(fatal.getCause.isInstanceOf[OutOfMemoryError],
+ s"the error wrapping a fatal one must reach the task as it is; got $fatal")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sortBy(_._1)}; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. Phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
+ val pushed = scanOf(df.queryExecution.executedPlan).simpleString(200)
+ assert(pushed.contains("GreaterThan(v,"),
+ s"the data filter must actually be pushed for this test to mean anything: $pushed")
+ // Storage filter on `k` (key column).
+ val withSF = scanWithStorageFilter(df, threshold = 100L)
+
+ val collected = executePlanCollect(withSF).toSet
+ // Every row satisfies v > 'v_000', so the storage filter alone decides the result.
+ val expected = rows.filter(_._1 >= 100L).toSet
+ assert(collected == expected,
+ s"got ${collected.toSeq.sortBy(_._1)}; expected ${expected.toSeq.sortBy(_._1)}")
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSet == Set((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty)
+ .getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSet == Set(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again, same rows, more bytes") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Two things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. And
+ // the fallback must actually have happened, which is observable. It reads the key column a
+ // second time, so it transfers strictly more bytes. Without that second assertion the test
+ // would pass even if the cap never reached the reader.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up past the cap still
+ // has to emit in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = kvSchema()
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") {
+ readCounting(path, formatReader(fileSchema, Nil, storageFilters, countingHadoopConf()))
+ }
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the
+ // row ranges. `k >= 350` keeps a contiguous run, so there is one range to hold.
+ val (plainRows, plainBytes) = run("100b")
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity = 16)
+ try {
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected a row group with no survivor at all; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group past the cap charges its second key read against the byte metric") {
+ // Giving splicing up means phase 2 reads the key columns a second time, while the baseline
+ // counts them once, in phase 1. That extra read is a cost against the saving, so the same file
+ // and filter must report less avoided than they do while splicing. Without that term the metric
+ // would credit the feature with bytes it did transfer.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = kvSchema()
+
+ def avoidedBytes(cap: Long, threshold: Long): Long = {
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))),
+ fileSchema, m, cap)
+ val (_, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => b.column(0).getLong(i), capacity = 16)
+ try m.bytesAvoidedByPageFiltering.value finally reader.close()
+ }
+
+ val spliced = avoidedBytes(64L * 1024 * 1024, 350L)
+ val plain = avoidedBytes(100L, 350L)
+ assert(spliced > 0, s"page filtering has to avoid something here; got $spliced")
+ assert(plain < spliced,
+ s"the second key read must count against the saving; plain=$plain spliced=$spliced")
+
+ // The two halves of the budget are weighed together, and this is the case that says so. Eight
+ // survivors of a long key buffer 8 * (1 + 8) = 72 bytes and fall in one range of 40, so
+ // neither half reaches the 100-byte cap on its own while the sum passes it at the seventh.
+ // Weighing them separately would keep splicing here, and report the larger saving for it.
+ val eightSpliced = avoidedBytes(64L * 1024 * 1024, 393L)
+ val eightPlain = avoidedBytes(100L, 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+ }
+ }
+
+ test("an accumulator sized up front is charged for its byte child when it is sized") {
+ // Every accumulator of a row group after the first sizes a variable-length key's byte child to
+ // what the one before it held, before any value arrives. Charged only as values arrive, the
+ // last accumulator of a row group would hold a whole accumulator's worth of memory for a few
+ // survivors' charge. Six survivors of a 1000-byte key over a capacity of 4 are charged about
+ // 24KB value by value, and about 32KB once the second accumulator's byte child is charged when
+ // it is sized. A cap between the two gives splicing up only in the second case, which the
+ // second read of the key column shows.
+ withTempDir { dir =>
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), REPEAT('x', 996)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val fileSchema = kvSchema(keyType = StringType)
+ val storageFilters = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "4") {
+ readCounting(path, formatReader(fileSchema, Nil, storageFilters, countingHadoopConf()))
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ val (cappedRows, cappedBytes) = run("28000b")
+ assert(splicedRows == 6 && cappedRows == 6,
+ s"the filter keeps keys 15..20 either way; got $splicedRows and $cappedRows")
+ assert(cappedBytes > splicedBytes,
+ "the second accumulator's byte child must count against the cap, so the row group gives " +
+ s"splicing up and reads the key column again; capped=$cappedBytes spliced=$splicedBytes")
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index does not cost the file its page filtering") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, and the
+ // reader answers that by reading no part of a row group for the rest of the file. The pushed
+ // data filter's page pruning survives that, so the rows would not show it. The WARN the
+ // exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(50L)))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(Array.fill(length)(0xFF.toByte))
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. Giving the row group up hands it to phase 2 whole, which reads the
+ // same pages in the same batches as a plain read, so the page fails at the same batch, after
+ // the same rows. That is what makes `ignoreCorruptFiles` keep exactly what it keeps of a plain
+ // read, and it is asserted here on the readers themselves, as the rows each hands back before
+ // it throws. The filter is selective, so a reader that kept applying it would return fewer.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ val outDir = new File(dir, "damaged").getAbsolutePath
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ rows.toDF("k", "v").repartition(1).write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 64L * 1024 * 1024)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .option(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT, 100L)
+ .parquet(outDir)
+ val path = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ .head.getAbsolutePath
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(false, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(false, 16, storageFilter)
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ (read.toSeq, error)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError) = readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry leaves the reader unable to narrow the rest of this file, since the next row
+ // group's store may be as unreadable as this one's. What it must not give up is the pushed data
+ // filter's own page pruning, which parquet did before the reader saw the block, and which a
+ // plain read of the same query keeps. Dropping that would read more than having no storage
+ // filter at all.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warned) = withPartialReadWarning(read(evenKeys))
+ assert(warned, "the read must have discovered the damaged index, or this test proves nothing")
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry has latched.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the splice queues and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = kvSchema()
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString), capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // Each survivor costs 9 bytes of long key and 40 of its own range. The sum passes the cap of
+ // 1024 at survivor 21, which releases the buffer, and the ranges alone at survivor 26, which
+ // gives the filter up. The capacity decides what the buffer holds when it is released. At
+ // 4096 that is one partly filled set of accumulators, and at 16 a full set in the queue as
+ // well. Both have to end with the queue empty and the filter given up. Keeping one without
+ // the other would leave phase 2 reading non-key columns while emit spliced keys from a queue
+ // holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString))
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSet == Set("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does, since the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers conjuncts under the relation's own column names") {
+ // The analyzer spells a reference the way the query did, so under case-insensitive resolution
+ // a bloom can name `_TMP_METADATA_ROW_INDEX` while the relation, and the schema the reader
+ // requests, call the column `_tmp_metadata_row_index`. The planner normalizes what it offers to
+ // the relation's names, so the format's gate sees the name the reader will see and declines.
+ // Without that the gate passed it, `ParquetStorageFilter.create` rejected it on the driver, and
+ // a query that runs with the conf off failed with it on.
+ withTempDir { dir =>
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", s"id AS $rowIndexName", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", rowIndexName, "v")
+ .queryExecution.optimizedPlan
+ def column(name: String): Attribute =
+ relation.output.find(_.name == name).getOrElse(fail(s"no $name in the relation output"))
+
+ def offered(key: Attribute): Seq[Expression] = {
+ val logical = LogicalFilter(
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ scanOf(physical).storageFilters
+ }
+
+ // As a control, an ordinary key spelled in another case is offered, under its own name.
+ val upperK = offered(column("k").withName("K"))
+ assert(upperK.size == 1, s"a bloom over `K` must reach the scan; got $upperK")
+ assert(upperK.head.references.map(_.name).toSet == Set("k"),
+ s"and it must carry the relation's name for the column; got ${upperK.head}")
+
+ val upperRowIndex =
+ offered(column(rowIndexName).withName(rowIndexName.toUpperCase(Locale.ROOT)))
+ assert(upperRowIndex.isEmpty,
+ s"a bloom over the row-index column must not be offered in any case; got $upperRowIndex")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for. A dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`, since `/` is floating-point division in Spark SQL, which would hand every
+ // key expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize, dictionary = dictionary)
+ }
+
+ // Every column chunk of the file, from its footer. The facts the tests below assert about a
+ // fixture are read off these rather than assumed from the write options.
+ private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] =
+ footerOf(filePath).getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq
+
+ private def footerOf(filePath: String): ParquetMetadata = {
+ val conf = spark.sessionState.newHadoopConf()
+ ParquetFooterReader.readFooter(
+ HadoopInputFile.fromPath(new Path(filePath), conf), ParquetMetadataConverter.NO_FILTER)
+ }
+
+ // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String, column: String = "k"): Set[Encoding] =
+ footerChunks(filePath).filter(_.getPath.toDotString == column)
+ .flatMap(_.getEncodings.asScala).toSet
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // builds the plain vectorized reader rather than the late-materialization one. Every read helper
+ // in this suite goes through here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need. The caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ if (storageFilter == null) {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } else {
+ new LateMaterializationParquetRecordReader(useOffHeap, capacity, storageFilter)
+ }
+ } { reader =>
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // A key value as a string in its *internal* representation, so the splicing path and the plain
+ // path can be compared without knowing the type and without external type conversion.
+ // `InternalRow.getAccessor` does the per-type read, which is why this takes the batch's row
+ // rather than the column vector.
+ private def renderValue(row: InternalRow, ordinal: Int, dt: DataType): String =
+ InternalRow.getAccessor(dt)(row, ordinal) match {
+ case null => "null"
+ // Two equal arrays are two objects, so the default toString would compare identities.
+ case bytes: Array[Byte] => bytes.mkString(",")
+ case value => value.toString
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path. Whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.getRow(i), 0, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage, one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why. There is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ } {
+ val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded"
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // This is a regression test. TimeType is an AtomicType that passes every planning-time gate
+ // (it is batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a
+ // TIME join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // What FileSourceStrategy offers, what ParquetStorageFilter.create admits and what the reader
+ // can copy are one list, because isSupportedKeyType answers from ValueCopier. Both are asserted
+ // here. A type the copier has no case for would be a task failure rather than a planning-time
+ // rejection, and the delegation is what rules that out.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ assert(ValueCopier.forType(dt) != null, s"$dt should have a value copier")
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ intercept[IllegalStateException](ValueCopier.forType(dt))
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, which fixes the key row's
+ // layout. The reader pairs each key column's batch slot with its key-row position, so nothing
+ // depends on that order. Every other fixture mentions its keys in column order, and production
+ // does not promise that, since the conjuncts arrive in `afterScanFilters` order, which follows
+ // the join. So this has two key columns mentioned the other way round, and asserts both the
+ // sorted layout and that every key still comes back with its own row.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // The emit path rewrites the key slots alone, and a partition slot is not one of them. It sits
+ // past the data columns and must keep the constant vector `initBatch` populated for it.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite. OffHeapColumnVector.close() frees
+ // the native buffer, so a read after close is a crash rather than stale data. A second close
+ // is harmless, since close() zeroes the addresses it freed. capacity = 16 over 71 survivors
+ // forces 5 emits, each closing the previous emit's dequeued key vectors. The other 29 rows
+ // are what makes a declining reader fail this.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(30L))
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (30L to 100L),
+ s"expected the 71 surviving keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // One ColumnarBatch is handed out for the whole read and its key slots are rewritten per
+ // batch, so a consumer holding on to an earlier getCurrentValue() would read the wrong
+ // vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new LateMaterializationParquetRecordReader(false, 8, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution, a column missing from the physical file -----
+
+ test("non-key column missing from a file: the evolved table reads and credits avoided bytes") {
+ // Schema evolution end to end, with the non-key column rather than the key missing from the
+ // older file. The clipped requested schema keeps the missing column, which the byte walks skip
+ // by themselves, since they go over the block's chunks and the older file has no chunk for it.
+ // What this pins is that such a read returns the right rows and still credits a saving.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // The older file has (k, v), the newer one (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSet
+ val expected = ((150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))).toSet
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // A positive total says the byte walks ran over the older file's row groups as well.
+ // Asserting non-negativity would prove nothing. `SQLMetric.add` drops a negative and the
+ // value cannot go below zero.
+ val bytesRg = withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value + bytesPf.value > 0,
+ s"the walk must credit some avoided bytes; got rg=${bytesRg.value} pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = scanOf(spark.read.parquet(path).select("k", "v").queryExecution.executedPlan)
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Set[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Set[(Long, Option[Long])]) = {
+ val withSF = withStorageFilters(spark.sql(s"SELECT id, k FROM $table")) { attr =>
+ Seq(buildFilter(attr("k")))
+ }
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSet
+ (withSF, collected)
+ }
+
+ test("all keys missing: an error on the constant reads the file rather than skipping it") {
+ // A file with none of the filter's key columns makes the predicate constant over it, so the
+ // reader decides once, from the values `initBatch` materialized for those columns. That
+ // evaluation can throw exactly as a row's value can. Here the older file's `k` reads as its
+ // DEFAULT, which is not a number, and the cast fails under ANSI. Failing open reads the file
+ // the way a plain scan would, so its rows survive. Skipping it would lose them, and rethrowing
+ // would fail a query a plain read answers.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true",
+ SQLConf.ANSI_ENABLED.key -> "true") {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql("ALTER TABLE evolved ADD COLUMN k STRING DEFAULT 'not-a-number'")
+ spark.sql("INSERT INTO evolved VALUES (4, '40'), (5, '50')")
+ val withSF = withStorageFilters(spark.sql("SELECT id, k FROM evolved")) { attr =>
+ Seq(GreaterThanOrEqual(Cast(attr("k"), LongType), Literal(1L)))
+ }
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val ids = collectRows(withSF).map(_.getLong(idPos)).toSet
+ assert(ids == Set(1L, 2L, 3L, 4L, 5L),
+ s"the older file must fail open and the newer one pass the cast; got $ids")
+ }
+ }
+ }
+
+ test("all keys missing: a checked exception on the constant reads the file") {
+ // The constant evaluation fails open under the same rule as a row's, a checked exception
+ // included. The older file's `k` reads as its DEFAULT 7, which the key expression throws on,
+ // so that file must be read the way a plain scan would. The newer file's keys evaluate
+ // normally and are rejected, since no Filter runs above this scan.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(ThrowsCheckedOn(k, bad = 7L), Literal(100L)))
+ assert(collected == Set((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L))),
+ s"the older file must fail open and the newer one be filtered; got $collected")
+ }
+ }
+ }
+
+ test("a file with none of the projected non-key columns is read plainly") {
+ // The clipped requested schema keeps a column the file does not have, so a projected non-key
+ // column added after the older file was written still looks like something for phase 2 to
+ // read. Phase 2 reads nothing for it, so for that file the filter could only add cost, and the
+ // reader declines it there. With no Filter above this scan, the older file's rows come back
+ // unfiltered while the newer file's are still filtered.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTable("evolved_value") {
+ spark.sql("CREATE TABLE evolved_value (k BIGINT, v STRING) USING parquet")
+ spark.sql("INSERT INTO evolved_value VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+ spark.sql("ALTER TABLE evolved_value ADD COLUMN w STRING")
+ spark.sql("INSERT INTO evolved_value VALUES (4, 'd', 'x'), (5, 'e', 'y')")
+ val withSF =
+ scanWithStorageFilter(spark.sql("SELECT k, w FROM evolved_value"), threshold = 5L)
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val keys = collectRows(withSF).map(_.getLong(kPos)).toSet
+ assert(keys == Set(1L, 2L, 3L, 5L),
+ s"the older file must be read unfiltered and the newer one filtered; got $keys")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test. The default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Set[(Long, Option[Long])] = Set((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(StorageFilterMetrics.ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Set((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got ${rejectsNull.toSeq.sorted}")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got ${acceptsNull.toSeq.sorted}")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ val collected = collectRows(withSF).map(r => (r.getLong(0), r.getLong(2))).toSet
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1)).toSet
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got " +
+ s"${collected.toSeq.sorted.take(5)}")
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF =
+ scanWithStorageFilter(spark.read.parquet(outDir).select("k", "s"), threshold = 195L)
+ val collected = collectRows(withSF)
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSet
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k")).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sorted}; expected $expected")
+ }
+ }
+ }
+
+ test("a storage filter binds to the right column with partition and metadata columns in output") {
+ // `preparedStorageFilters` binds the filter's references against `output.take(requiredSchema
+ // .length)`, and `output` is the data columns, then the generated metadata columns, then the
+ // partition columns, then the constant metadata ones. A projection carrying all four is where a
+ // wrong prefix would bind the predicate to another column and drop rows the filter keeps, which
+ // the post-scan Filter cannot put back. The file's first column `a` is not projected, so the
+ // key sits at a different position in the relation's data schema than in the scan's.
+ withTempDir { dir =>
+ val outDir = new File(dir, "parted").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id * 7 AS a", "id AS k", "CONCAT('v_', id) AS v", "CAST(id % 3 AS INT) AS p")
+ .repartition(1)
+ .write
+ .partitionBy("p")
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select(
+ col("k"), col("v"), col("p"),
+ col("_metadata.row_index").as("ri"), col("_metadata.file_size").as("fs"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ // The assumption the binding rests on, stated where a change to `output` would break it.
+ assert(withSF.output.take(withSF.requiredSchema.length).map(_.name) ==
+ withSF.requiredSchema.fieldNames.toSeq,
+ s"the scan's leading attributes must be its data columns; got " +
+ s"${withSF.output.map(_.name)}")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val pPos = withSF.output.indexWhere(_.name == "p")
+ def collectAll(plan: SparkPlan): Set[Seq[Any]] =
+ collectRows(plan).map(_.toSeq(plan.output.map(_.dataType))).toSet
+ val filtered = collectAll(withSF)
+ // The same scan without the filter is the oracle, so the row indexes and file sizes come
+ // from the read rather than from arithmetic over how the partitions were written.
+ val expected = collectAll(withSF.copy(storageFilters = Nil))
+ .filter(_(kPos).asInstanceOf[Long] >= 150L)
+ assert(filtered == expected,
+ s"expected ${expected.size} rows with their own partition and metadata values; " +
+ s"got ${filtered.size}")
+ assert(filtered.map(_(pPos)).size == 3,
+ s"every partition value must still be represented; got ${filtered.map(_(pPos))}")
+ }
+ }
+ }
+
+ // ----- Planner gates -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false. It may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ //
+ // This hook answers about shapes and types alone. Whether the conjunct is deterministic and
+ // whether its references are projected columns of this scan are plan-visible, so they stay with
+ // the planner; `FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter`
+ // covers that half.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ // A cast key is supported. In ANSI mode it can throw on a row an earlier conjunct would have
+ // dropped, and the reader handles that where it arises, by giving the filter up for the row
+ // group. Declining here instead would also decline every widening cast, which is what type
+ // coercion inserts for a join between an int and a bigint column.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+
+ // The conf is read in the format rather than in the planner, and it is the per-scan question
+ // that carries it, asked once before anything per conjunct.
+ assert(!format.supportsStorageFilterPushdown(spark), "the feature is off with the conf off")
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ assert(format.supportsStorageFilterPushdown(spark), "and on with it on")
+ assert(!subclass.supportsStorageFilterPushdown(spark), "a subclass must not claim support")
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ assert(format.supportsStorageFilter(onCast),
+ "a cast key is pushed, and an evaluation error gives the row group up")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The rows are right either way, since the bloom stays in the post-scan Filter. What this gate
+ // saves is offering the scan a filter its reader cannot apply.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one. Every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSet
+ assert(keys == (1L to 50L).toSet,
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support declines rather than delegating") {
+ // The default `FileFormat.buildReaderWithStorageFilters` answers None, and the caller falls
+ // back to the ordinary builder. That is what keeps the two builders from being able to call
+ // each other, and declining is safe because the post-scan Filter still holds the conjunct.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(1L)))
+ val declined = new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration(), Map.empty)
+ assert(declined.isEmpty, "the default must not build a reader of its own")
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilterPushdown(spark),
+ "and it must not claim support either")
+ }
+
+ test("a scan whose format declines falls back to the ordinary reader and returns every row") {
+ // The check above is that the default answers None. This is the other half, that `inputRDD`
+ // then builds the ordinary reader and the scan still produces its rows. A `ParquetFileFormat`
+ // subclass is the shape that reaches it, since it inherits the override and declines by class,
+ // and its ordinary builder is the one an extension would have customized.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, threshold = 195L)
+ val declining = withSF.copy(
+ relation = withSF.relation.copy(fileFormat = new ParquetFileFormat() {})(spark))
+ val collected = executePlanCollect(declining).toSet
+ assert(collected == rows.toSet,
+ s"a declining format reads the file plainly; got ${collected.size} of ${rows.size} rows")
+ }
+ }
+ }
+
+ Seq(false, true).foreach { aqe =>
+ test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") {
+ // AQE is on by default in production, and it is where the bloom subquery is planned by
+ // PlanAdaptiveSubqueries rather than PlanSubqueries, which is the path
+ // preparedStorageFilters' ScalarSubquery materialization depends on.
+ withBloomFilterTables {
+ val baseConf = Map(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString)
+ def run(pushdown: Boolean): (Int, Set[(Long, Long)]) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, rows) = runBloomFilterJoin()
+ (countBloomFiltersInStorageFilters(plan),
+ rows.map(r => (r.getLong(0), r.getLong(1))).toSet)
+ }
+ val (offFilters, off) = run(false)
+ val (onFilters, on) = run(true)
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ // Equal results are guaranteed by the post-scan Filter whether or not the scan applied the
+ // filter, so they alone would not notice AQE dropping it on the way to the final plan.
+ assert(offFilters == 0 && onFilters == 1,
+ s"the scan must carry the filter with the conf on and not with it off; " +
+ s"got on=$onFilters off=$offFilters")
+ }
+ }
+ }
+
+ test("canonicalization keeps a storage-filter scan distinct from a plain one") {
+ // storageFilters is in doCanonicalize and in the case class equality, which is what stops
+ // exchange and subquery reuse from serving one scan's result to the other. Reuse compares
+ // canonicalized plans, so a scan that filters must not match a plain scan of the same file, and
+ // two scans carrying the same filter must still match.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, threshold = 25L)
+ val plain = withSF.copy(storageFilters = Nil)
+ assert(withSF != plain, "case class equality must take storageFilters into account")
+ assert(!withSF.sameResult(plain),
+ s"a filtering scan must not be reusable as a plain one:\n${withSF.canonicalized}\n" +
+ s"${plain.canonicalized}")
+ // A second, independently planned scan of the same file with the same filter still
+ // matches, so reuse is not disabled wholesale. Its key attribute carries a different
+ // exprId, which is what canonicalization normalizes away.
+ val sameSF = scanWithStorageFilter(path, threshold = 25L)
+ assert(withSF.sameResult(sameSF),
+ s"two scans with the same storage filter must stay reusable:\n" +
+ s"${withSF.canonicalized}\n${sameSF.canonicalized}")
+ val otherSF = scanWithStorageFilter(path, threshold = 30L)
+ assert(!withSF.sameResult(otherSF), "a different threshold is a different result")
+ }
+ }
+ }
+
+ test("whole-stage codegen off: the row-at-a-time path still applies the extracted bloom") {
+ // The planner's extraction gate is `fileFormat.supportBatch`, which does not look at
+ // whole-stage codegen, while `FileSourceScanExec.supportsColumnar` does. So with codegen off
+ // the bloom is still extracted, `returningBatch` is false, and the reader serves the spliced
+ // batch one row at a time. That is the only combination where the planner's gate is weaker
+ // than the runtime's.
+ withBloomFilterTables {
+ val baseConf = Map(
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "false")
+ def run(pushdown: Boolean): (Set[(Long, Long)], Int, Boolean, Long) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, result) = runBloomFilterJoin()
+ val scans = plan.collect { case s: FileSourceScanExec => s }
+ val excluded = scans.filter(_.storageFilters.nonEmpty).map { s =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+ (result.map(r => (r.getLong(0), r.getLong(1))).toSet,
+ countBloomFiltersInStorageFilters(plan), scans.forall(!_.supportsColumnar), excluded)
+ }
+ val (off, _, _, _) = run(false)
+ val (on, storageBlooms, noColumnarScan, excluded) = run(true)
+ assert(storageBlooms >= 1, s"the bloom must still be extracted with codegen off; " +
+ s"got $storageBlooms")
+ assert(noColumnarScan, "with codegen off no scan should output columnar batches")
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ // The assertions above hold whether or not the row path honored the filter, since the
+ // post-scan Filter answers the query either way. The metrics are what say it did.
+ assert(excluded > 0, s"the row path must have excluded rows; the metrics say $excluded")
+ }
+ }
+
+ test("off-heap column vectors through the planner: spliced values survive the free") {
+ // Off-heap is where the vector lifecycle actually bites. The previous batch's key vectors are
+ // freed at the next nextBatch(), so a stale reference reads released native memory rather than
+ // old bytes. This drives it through the planner, where the batch also crosses
+ // ColumnarToRowExec.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.COLUMN_VECTOR_OFFHEAP_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 150L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 150L).toSet
+ assert(collected == expected,
+ s"got ${collected.size} rows; expected ${expected.size}. " +
+ s"first few: ${collected.toSeq.sortBy(_._1).take(3)}")
+ }
+ }
+ }
+
+ // ----- Page-level pushedFilterRanges (a strict subset of the row group) -----
+
+ test("pushed data filter narrows to a page subset: phase 1 stays aligned with the row indexes") {
Review Comment:
Nit: this runs the same fixture and path as "page-subset ranges still credit the avoided bytes" (L2374): one row group of 100-row pages, a pushed `v` range that drops leading pages, a `k >= t` storage filter whose survivors sit in the last page, and an exact-rows assertion. Round 6 gave that test a premise check (`pushedOnly < rows.size`, L2389-2391). This one only checks that the filter string reached the scan (L2361), so if narrowing stopped it would pass without testing any alignment. `r._2 >= "v_0300"` at L2366 is also always true for `k >= 350`. Could its rationale move into the other test, and this one go?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-6033739696
@dongjoon-hyun @sunchao thank you for the approvals and the reviews. All 7 of dongjoon's items and sunchao's first are done in one commit, and the second is a follow-up in the description. The conf doc now says it is the V1 file source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4192664268
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1084 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A byte child sized up front is
+ * charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}. A
+ * fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // decoders of their last page.
+ closeDataPages();
+ releaseRowGroupReaders();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (reportedMessages.add(OFFSET_INDEX_UNREADABLE)) {
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn(OFFSET_INDEX_UNREADABLE, MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long survivors = 0L;
+ long firstSurvivorAt = 0L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_DECODING, true);
+ // The failed decode may have grown the scratch vectors, and phase 2 reads the same pages
+ // into vectors of its own. So capacity-sized ones replace them, and beside a plain read's
+ // own vectors the reader then holds one capacity-sized batch of key vectors.
+ closeAll(keyScratchVectors);
+ allocateKeyScratch();
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = predicate.eval(keyScratchBatch.getRow(r));
+ } catch (Throwable t) {
+ // Fall back on anything but a task kill or a fatal error. The predicate runs here without
+ // the conjuncts before it in the plan, so it can throw on a row a plain scan never
+ // evaluates it on (see `ParquetStorageFilter.isSupportedStorageFilter`). Giving the
+ // filter up returns every row the pushed filter kept, and the post-scan Filter decides
+ // them in its own order, raising the same error for a row it does evaluate.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK exception
+ // (`timestamp_seconds` of a decimal) or an `AssertionError`, and a Hive UDF throws a
+ // checked SparkException. Rethrowing would fail a query a plain read answers, or drop
+ // the rest of the file silently under `ignoreCorruptFiles`.
+ //
+ // A fatal error goes out even on a row an earlier conjunct of the plan would have
+ // rejected, so the reader never runs on after one the executor would take as fatal.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_EVALUATION, false);
+ return null;
+ }
+ if (!survives) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ long rowsBefore = pushedFilterRowCount - remaining + r;
+ if (survivors++ == 0) firstSurvivorAt = rowsBefore;
+ if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ if (currentKeyAccumulators == null) {
+ // The survivors from this one on, at the rate seen since the first one, so that rows
+ // before it, which a clustered key leaves, do not dilute the rate.
+ long expected = (long) Math.ceil(
+ (double) survivors * (remaining - r) / (rowsBefore - firstSurvivorAt + 1));
+ splicedBytes += allocateKeyAccumulators(cap - splicedBytes - rangeBytes, expected);
+ }
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // The buffer is this reader's own and optional, so failing to fill it gives splicing up
+ // rather than the filter, and the memory goes back at once. What fails here:
+ // - a survivor vector that cannot grow;
+ // - a key value the predicate did not read on this row, such as `b` in
+ // `coalesce(a, b)`, since the copy decodes every key;
+ // - a bug of ours, which only the WARN then shows.
+ // Phase 2 reads the key columns the way a plain read does, so a value that cannot be
+ // decoded fails the read only where a plain read fails.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together.
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, and returns the charge for the
+ * byte children it sized up front. The rest of the set is charged per row as survivors arrive.
+ * A variable-length key's byte child is reserved for what the set queued before it held, scaled
+ * to the {@code expectedRows} survivors this set can expect, where that is more than its default
+ * allocation. So it does not grow one doubling at a time for every capacity's worth of
+ * survivors, and a sparse last set is not sized as a full one. With its default doubling,
Review Comment:
Thanks, agreed. With the up-front sizing gone, as you suggest in the next thread, the rate and this comment are gone too.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1084 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ * The slots are swapped rather than copied into the persistent vectors on purpose. A copy would
+ * pay a second value copy per surviving row, a byte copy for a variable-length key, where the
+ * swap is one array write per key column per batch.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows, by default. A factor rather than an exact figure, so with that default this
+ * half of the charge over-counts rather than under-counts, since nothing above this buffer can
+ * spill it. A {@code hugeVectorReserveRatio} above 2 makes it under-count for a child past
+ * {@code hugeVectorThreshold}. That threshold is off by default. A byte child sized up front is
+ * charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}. A
+ * fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages this reader has reported, each once per split. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION, false);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column. A struct
+ // the file has, but none of whose requested fields it has, counts only under the legacy
+ // `returnNullStructIfAllFieldsMissing`, since otherwise the clipped schema reads one of its
+ // other fields to tell a null struct from one whose requested fields are all null.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus the
+ // `DEFAULT_CHILD_BYTES_PER_ROW` the child is allocated per row, and a null byte for each.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow +=
+ variableLength ? 1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive. Its column readers go
+ // too. They hold the row ranges it was read over, which the budget no longer counts, and the
+ // decoders of their last page.
+ closeDataPages();
+ releaseRowGroupReaders();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (reportedMessages.add(OFFSET_INDEX_UNREADABLE)) {
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn(OFFSET_INDEX_UNREADABLE, MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long survivors = 0L;
+ long firstSurvivorAt = 0L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_DECODING, true);
+ // The failed decode may have grown the scratch vectors, and phase 2 reads the same pages
+ // into vectors of its own. So capacity-sized ones replace them, and beside a plain read's
+ // own vectors the reader then holds one capacity-sized batch of key vectors.
+ closeAll(keyScratchVectors);
+ allocateKeyScratch();
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = predicate.eval(keyScratchBatch.getRow(r));
+ } catch (Throwable t) {
+ // Fall back on anything but a task kill or a fatal error. The predicate runs here without
+ // the conjuncts before it in the plan, so it can throw on a row a plain scan never
+ // evaluates it on (see `ParquetStorageFilter.isSupportedStorageFilter`). Giving the
+ // filter up returns every row the pushed filter kept, and the post-scan Filter decides
+ // them in its own order, raising the same error for a row it does evaluate.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK exception
+ // (`timestamp_seconds` of a decimal) or an `AssertionError`, and a Hive UDF throws a
+ // checked SparkException. Rethrowing would fail a query a plain read answers, or drop
+ // the rest of the file silently under `ignoreCorruptFiles`.
+ //
+ // A fatal error goes out even on a row an earlier conjunct of the plan would have
+ // rejected, so the reader never runs on after one the executor would take as fatal.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_EVALUATION, false);
+ return null;
+ }
+ if (!survives) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ long rowsBefore = pushedFilterRowCount - remaining + r;
+ if (survivors++ == 0) firstSurvivorAt = rowsBefore;
+ if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ if (currentKeyAccumulators == null) {
+ // The survivors from this one on, at the rate seen since the first one, so that rows
+ // before it, which a clustered key leaves, do not dilute the rate.
+ long expected = (long) Math.ceil(
+ (double) survivors * (remaining - r) / (rowsBefore - firstSurvivorAt + 1));
+ splicedBytes += allocateKeyAccumulators(cap - splicedBytes - rangeBytes, expected);
+ }
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // The buffer is this reader's own and optional, so failing to fill it gives splicing up
+ // rather than the filter, and the memory goes back at once. What fails here:
+ // - a survivor vector that cannot grow;
+ // - a key value the predicate did not read on this row, such as `b` in
+ // `coalesce(a, b)`, since the copy decodes every key;
+ // - a bug of ours, which only the WARN then shows.
+ // Phase 2 reads the key columns the way a plain read does, so a value that cannot be
+ // decoded fails the read only where a plain read fails.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together.
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, and returns the charge for the
+ * byte children it sized up front. The rest of the set is charged per row as survivors arrive.
+ * A variable-length key's byte child is reserved for what the set queued before it held, scaled
+ * to the {@code expectedRows} survivors this set can expect, where that is more than its default
+ * allocation. So it does not grow one doubling at a time for every capacity's worth of
+ * survivors, and a sparse last set is not sized as a full one. With its default doubling,
+ * {@code reserve} takes twice what it is asked for, so with its null bytes the child takes the
+ * factor's four bytes per value byte. That happens only when the charge fits within
+ * {@code headroom}, what the cap leaves, together with the fixed part and a range of each of
+ * those rows, since a scattered survivor adds a range. So the rows the set is sized for stay
+ * within the cap. The sizing is a prediction, though. A set sized for values longer than the
+ * ones that then arrive keeps the unused part charged, since its child holds that memory. So a
+ * larger cap can still give splicing up where a smaller one, which grew the set instead, keeps
+ * splicing. That costs one more read of the key columns, never a different answer. The memory
+ * is taken before any value arrives, so it is charged now, as if those values had arrived, and
+ * the values written into it use that charge up before they add to it.
+ */
+ private long allocateKeyAccumulators(long headroom, long expectedRows) {
Review Comment:
Done, thanks. Every set now grows the way a row group's first set does and is charged per value, so what a row group is charged no longer depends on the cap. That removes `prepaidValueBytes`, the rate and the headroom check, and the fixed-part and sparse-last-set tests with them. The 32142 test now pins the per-value charge, and the scattered one checks that every cap from its charge, 24222, up splices, and that the second set's byte child grows to 2000 bytes where sizing would have reserved 8000. It brings back the regrowth copies your R6 item measured, for a variable-length key's sets after a row group's first one. The javadoc now says so, and why a set is not sized.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3742 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t: VirtualMachineError => fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the survivors it can expect and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the survivors it can still expect at the rate seen since the row group's
+ // first survivor. That memory is taken before any value arrives, so it is charged then. Six
+ // survivors over a capacity of 4 fill one set of four 1000-byte keys, and the second set starts
+ // two rows before the end of the row group, with every row since the first survivor kept. So
+ // it is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000
+ // bytes each, so they add 8000 more once the sizing is used up, and the row group holds 32142
+ // bytes in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or
+ // a double charge would each get wrong. The second set's byte child is reserved for 2000
+ // bytes, which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000.
+ // And the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ assertSpliceCharge(32142L, rows = 6) { cap =>
+ readUnderCap(path, laterSetSchema, laterSetFilter, cap, batchSize = 4)
+ }
+
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ if (cap >= 32142L) {
+ assert(secondSet == Seq(4000),
+ s"the second set must be sized for the two values left under cap $cap; got $secondSet")
+ }
+ }
+ }
+ }
+
+ test("a later accumulator set is sized only when it fits with its rows' fixed part and ranges") {
+ // The second set's two keys are 500 bytes here, shorter than the first set's 1000, so growing
+ // value by value is the cheaper path: 20142 bytes in all against 24142 sized. A cap of 24222
+ // sizes the set, whose byte child is then reserved for 2000 bytes and holds 4000. One of
+ // 24221 leaves 8113 bytes after the first set and its range, a byte short of the 8000 of
+ // sizing, the 34 of the two rows' fixed part and the 80 of a range for each. So the set grows
+ // instead, to a 1000-byte child, and keeps splicing. A cap of 20141 gives splicing up.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 500)
+ def run(maxSplicedBytes: String): Long =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)._2
+
+ val uncapped = run("64MB")
+ assert(run("24222b") == uncapped, "a cap that fits the sizing must keep splicing")
+ assert(run("24221b") == uncapped, "and so must one that only fits growing value by value")
+ assert(run("20141b") > uncapped, "a cap under both must give splicing up")
+
+ Seq(Long.MaxValue -> 4000, 24222L -> 4000, 24221L -> 1000).foreach { case (cap, child) =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected, s"every survivor must come back with its own key under cap $cap")
+ assert(secondSet == Seq(child),
+ s"under cap $cap the second set's byte child must hold $child bytes; got $secondSet")
+ }
+ }
+ }
+
+ test("a later set is sized only when the ranges of its scattered survivors fit too") {
+ // Keys 9 to 12 fill the first set over a capacity of 4, and keys 14 and 17 come scattered, each
+ // in a range of its own. The second set opens at key 14, sized for the four rows the rate
+ // since the first survivor promises, and its sizing is charged up front. So it is sized only
+ // when the cap holds a range for each of those rows as well. Without that, caps of 32216 to
+ // 32221 sized the set and then passed the cap on key 17's range, giving splicing up where a
+ // smaller cap of 32215, which grows the set instead, kept it. For this file, every cap from
+ // the smallest that splices up must splice, and every cap returns the same six rows.
+ withTempDir { dir =>
+ val (path, _) = writeLaterSetFile(dir, lastTwoBytes = 1000)
+ val prefix = Substring(BoundReference(0, StringType, nullable = true), Literal(1), Literal(4))
+ val keys = Seq("0009", "0010", "0011", "0012", "0014", "0017").map(Literal(_))
+ def readUnder(cap: Long): (Int, Long) =
+ readUnderCap(path, laterSetSchema, Seq(In(prefix, keys)), s"${cap}b", batchSize = 4)
+ val (_, uncapped) = readUnder(64L * 1024 * 1024)
+ val (smallerRows, smaller) = readUnder(24221L)
+ assert(smallerRows == 6 && smaller > uncapped, "the smallest cap that splices must be 24222")
+ Seq(24222L, 32215L, 32216L, 32221L, 32222L).foreach { cap =>
+ val (rows, bytes) = readUnder(cap)
+ assert(rows == 6 && bytes == uncapped,
+ s"a cap of $cap must keep the six rows and splicing; got $rows rows")
+ }
+ }
+ }
+
+ test("a sparse last set is sized for the survivors it can expect") {
+ // Every tenth of 125 keys survives, so over a capacity of 4 the fourth set holds key 120 alone.
+ // It opens five rows before the end of the row group, at a rate of 13 survivors in 121 rows, so
+ // it expects one survivor and is sized for one 1000-byte value, which `reserve` doubles to a
+ // 2000-byte child. Sized for the five rows left, as a full set, it would have taken 8000. The
+ // two sets before it are sized as full ones, since the rate promises them four survivors each.
+ withTempDir { dir =>
+ val df = spark.range(0, 125).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), REPEAT('x', 996)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 1024L * 1024)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val lastDigit =
+ Substring(BoundReference(0, StringType, nullable = true), Literal(4), Literal(1))
+ val (rows, reader) = readAllWith(path, Seq("k", "v"),
+ createFilter(Seq(EqualTo(lastDigit, Literal("0"))), laterSetSchema),
+ (b, i) => (b.column(0).getUTF8String(i).toString.take(4),
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ assert(rows.map(_._1) == (0 to 120 by 10).map(id => f"$id%04d"), s"got ${rows.map(_._1)}")
+ val children = rows.map(_._2)
+ assert(children.slice(4, 12).distinct == Seq(8000) && children.drop(12) == Seq(2000),
Review Comment:
This test is gone with the sizing.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3742 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t: VirtualMachineError => fail(s"the reader must fall back on $e; got $t", t)
Review Comment:
Done, it catches `LinkageError` too.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import scala.util.control.NonFatal
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while a code-generated `created`
+ // holds the deserialized filter instead. Dropping the expression lets a columnar scan's task
+ // reclaim those bytes for the rest of its life. An interpreted `created`, the fallback when
+ // codegen fails or is off, keeps the expression itself, so the bytes stay. A row-based scan
+ // keeps them anyway, since its closure captures the scan node, whose storage filters hold the
+ // same bytes.
+ // Safe because every task attempt deserializes its own copy from the driver's bytes, and this
+ // lazy val is the only reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while applying a storage filter, if it must not fall back on
+ * it by giving the filter up. `fromRead` says the error came from decoding the key pages rather
+ * than from evaluating the filter. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped when the error is one the executor's kill case takes,
+ * see [[throwIfKilled]]. The error may be the kill's interrupt surfacing through a UDF that
+ * wrapped it. `PythonRunner` reads a kill off the task context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had a plain plan raised `e`, goes out as itself, so the executor finds
+ * it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in an `UnknownError`, a fatal error of another type.
+ * `FileScanRDD` would take a bare `InternalError` from a reader for a corrupt file under
+ * `ignoreCorruptFiles`. A `NonFatal` wrapper does not work either. With a kill pending, the
+ * executor reports one as killed and keeps running. A plain plan raises it from the post-scan
+ * Filter, where neither happens, and the executor stops. It stops on the `UnknownError` too, at
+ * any `spark.executor.killOnFatalError.depth` of 1 or more, since that is at the top.
+ *
+ * An error from decoding differs in three ways:
+ * - a plain read raises it from the reader, where `FileScanRDD` wraps a `NonFatal` one in
+ * FAILED_READ_FILE before the executor looks. So a fatal error deeper in the chain of a
+ * `NonFatal` one has to sit one level shallower to count, while any other error is searched
+ * as deep as the executor does. Under `ignoreCorruptFiles` a plain read may skip the rest of
+ * the file on it instead.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ *
+ * Anything else falls back. An evaluation error is the filter's own, which the post-scan Filter
+ * raises again for a row it does evaluate. A decoding error is met again by the plain read of
+ * the same pages.
+ */
+ def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new UnknownError("A storage filter met a fatal error it must not absorb")
+ .initCause(internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ val depth = if (fromRead && NonFatal(e)) fatalErrorDepth - 1 else fatalErrorDepth
+ if (Executor.isFatalError(e, depth)) {
+ // The one `Executor.isFatalError` found, which no SparkOutOfMemoryError can precede.
+ val fatal = Iterator.iterate(e)(_.getCause).find(Utils.isFatalError).get
+ if (!corrupt(fatal)) throw out(fatal)
+ }
+ }
+
+ /**
+ * Whether `e` is how `WritableColumnVector.reserve` reports a vector that ran out of memory to
+ * grow, a plain `RuntimeException` caused by an `OutOfMemoryError`. The one it raises with no
+ * cause, for a capacity past what a vector can hold, has no fatal error to find, so the reader
+ * falls back on it anyway.
+ */
+ private def isVectorGrowthFailure(e: Throwable): Boolean =
+ e.getClass == classOf[RuntimeException] && e.getCause.isInstanceOf[OutOfMemoryError]
+
+ /**
+ * Throws a pending task kill, for the reader's loops that can run long without returning to
+ * `FileScanRDD`, which is where a plain read meets one.
+ */
+ def throwIfKilled(): Unit = throwIfKilled(null)
+
+ /**
+ * Throws a pending task kill, wrapped in a checked [[SparkException]] around `surfaced`, the
+ * error the kill surfaced as, if any. Under `ignoreCorruptFiles`, `FileScanRDD` would log any
+ * `RuntimeException` or `IOException` from a reader as a corrupt file and go on to open the
+ * split's remaining files, since it checks for a kill only between batches. The executor reports
+ * the task as killed either way. This covers the kills this reader sees at its own checks and
+ * catches. One that surfaces as an `IOException` from `readFilteredRowGroup` or from phase 2's
+ * decoding reaches that catch as it does for a plain read.
+ *
+ * The executor reports a task as killed only for an `InterruptedException` or a `NonFatal`
+ * error. So a `surfaced` error of any other type goes out as it is, the way a plain read raises
+ * it when decoding meets the same error. For example a codec's `InternalError`, or a native
+ * codec's `UnsatisfiedLinkError`. One from evaluating the filter goes out as it is too. Only a
+ * built-in expression can raise one there, such as an `ExceptionInInitializerError`, since
Review Comment:
Thanks, right. The sentence keeps the behaviour and the `ExceptionInInitializerError` example, and the reason is gone.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import scala.util.control.NonFatal
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while a code-generated `created`
+ // holds the deserialized filter instead. Dropping the expression lets a columnar scan's task
+ // reclaim those bytes for the rest of its life. An interpreted `created`, the fallback when
+ // codegen fails or is off, keeps the expression itself, so the bytes stay. A row-based scan
+ // keeps them anyway, since its closure captures the scan node, whose storage filters hold the
+ // same bytes.
+ // Safe because every task attempt deserializes its own copy from the driver's bytes, and this
+ // lazy val is the only reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while applying a storage filter, if it must not fall back on
+ * it by giving the filter up. `fromRead` says the error came from decoding the key pages rather
+ * than from evaluating the filter. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped when the error is one the executor's kill case takes,
+ * see [[throwIfKilled]]. The error may be the kill's interrupt surfacing through a UDF that
+ * wrapped it. `PythonRunner` reads a kill off the task context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had a plain plan raised `e`, goes out as itself, so the executor finds
+ * it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in an `UnknownError`, a fatal error of another type.
+ * `FileScanRDD` would take a bare `InternalError` from a reader for a corrupt file under
+ * `ignoreCorruptFiles`. A `NonFatal` wrapper does not work either. With a kill pending, the
+ * executor reports one as killed and keeps running. A plain plan raises it from the post-scan
+ * Filter, where neither happens, and the executor stops. It stops on the `UnknownError` too, at
+ * any `spark.executor.killOnFatalError.depth` of 1 or more, since that is at the top.
+ *
+ * An error from decoding differs in three ways:
+ * - a plain read raises it from the reader, where `FileScanRDD` wraps a `NonFatal` one in
+ * FAILED_READ_FILE before the executor looks. So a fatal error deeper in the chain of a
+ * `NonFatal` one has to sit one level shallower to count, while any other error is searched
+ * as deep as the executor does. Under `ignoreCorruptFiles` a plain read may skip the rest of
+ * the file on it instead.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ *
+ * Anything else falls back. An evaluation error is the filter's own, which the post-scan Filter
+ * raises again for a row it does evaluate. A decoding error is met again by the plain read of
+ * the same pages.
+ */
+ def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new UnknownError("A storage filter met a fatal error it must not absorb")
+ .initCause(internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ val depth = if (fromRead && NonFatal(e)) fatalErrorDepth - 1 else fatalErrorDepth
+ if (Executor.isFatalError(e, depth)) {
+ // The one `Executor.isFatalError` found, which no SparkOutOfMemoryError can precede.
+ val fatal = Iterator.iterate(e)(_.getCause).find(Utils.isFatalError).get
+ if (!corrupt(fatal)) throw out(fatal)
+ }
+ }
+
+ /**
+ * Whether `e` is how `WritableColumnVector.reserve` reports a vector that ran out of memory to
+ * grow, a plain `RuntimeException` caused by an `OutOfMemoryError`. The one it raises with no
+ * cause, for a capacity past what a vector can hold, has no fatal error to find, so the reader
+ * falls back on it anyway.
+ */
+ private def isVectorGrowthFailure(e: Throwable): Boolean =
+ e.getClass == classOf[RuntimeException] && e.getCause.isInstanceOf[OutOfMemoryError]
+
+ /**
+ * Throws a pending task kill, for the reader's loops that can run long without returning to
+ * `FileScanRDD`, which is where a plain read meets one.
+ */
+ def throwIfKilled(): Unit = throwIfKilled(null)
+
+ /**
+ * Throws a pending task kill, wrapped in a checked [[SparkException]] around `surfaced`, the
+ * error the kill surfaced as, if any. Under `ignoreCorruptFiles`, `FileScanRDD` would log any
+ * `RuntimeException` or `IOException` from a reader as a corrupt file and go on to open the
+ * split's remaining files, since it checks for a kill only between batches. The executor reports
+ * the task as killed either way. This covers the kills this reader sees at its own checks and
+ * catches. One that surfaces as an `IOException` from `readFilteredRowGroup` or from phase 2's
+ * decoding reaches that catch as it does for a plain read.
+ *
+ * The executor reports a task as killed only for an `InterruptedException` or a `NonFatal`
+ * error. So a `surfaced` error of any other type goes out as it is, the way a plain read raises
+ * it when decoding meets the same error. For example a codec's `InternalError`, or a native
Review Comment:
Done, both. The kill sentence now says it is how a plain read raises a decoding error, and the follow-up no longer mentions sizing.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3742 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t: VirtualMachineError => fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the survivors it can expect and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the survivors it can still expect at the rate seen since the row group's
+ // first survivor. That memory is taken before any value arrives, so it is charged then. Six
+ // survivors over a capacity of 4 fill one set of four 1000-byte keys, and the second set starts
+ // two rows before the end of the row group, with every row since the first survivor kept. So
+ // it is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000
+ // bytes each, so they add 8000 more once the sizing is used up, and the row group holds 32142
+ // bytes in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or
+ // a double charge would each get wrong. The second set's byte child is reserved for 2000
+ // bytes, which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000.
+ // And the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ assertSpliceCharge(32142L, rows = 6) { cap =>
+ readUnderCap(path, laterSetSchema, laterSetFilter, cap, batchSize = 4)
+ }
+
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ if (cap >= 32142L) {
+ assert(secondSet == Seq(4000),
+ s"the second set must be sized for the two values left under cap $cap; got $secondSet")
+ }
+ }
+ }
+ }
+
+ test("a later accumulator set is sized only when it fits with its rows' fixed part and ranges") {
+ // The second set's two keys are 500 bytes here, shorter than the first set's 1000, so growing
+ // value by value is the cheaper path: 20142 bytes in all against 24142 sized. A cap of 24222
+ // sizes the set, whose byte child is then reserved for 2000 bytes and holds 4000. One of
+ // 24221 leaves 8113 bytes after the first set and its range, a byte short of the 8000 of
+ // sizing, the 34 of the two rows' fixed part and the 80 of a range for each. So the set grows
+ // instead, to a 1000-byte child, and keeps splicing. A cap of 20141 gives splicing up.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 500)
+ def run(maxSplicedBytes: String): Long =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)._2
+
+ val uncapped = run("64MB")
+ assert(run("24222b") == uncapped, "a cap that fits the sizing must keep splicing")
+ assert(run("24221b") == uncapped, "and so must one that only fits growing value by value")
+ assert(run("20141b") > uncapped, "a cap under both must give splicing up")
+
+ Seq(Long.MaxValue -> 4000, 24222L -> 4000, 24221L -> 1000).foreach { case (cap, child) =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected, s"every survivor must come back with its own key under cap $cap")
+ assert(secondSet == Seq(child),
+ s"under cap $cap the second set's byte child must hold $child bytes; got $secondSet")
+ }
+ }
+ }
+
+ test("a later set is sized only when the ranges of its scattered survivors fit too") {
+ // Keys 9 to 12 fill the first set over a capacity of 4, and keys 14 and 17 come scattered, each
+ // in a range of its own. The second set opens at key 14, sized for the four rows the rate
+ // since the first survivor promises, and its sizing is charged up front. So it is sized only
+ // when the cap holds a range for each of those rows as well. Without that, caps of 32216 to
+ // 32221 sized the set and then passed the cap on key 17's range, giving splicing up where a
+ // smaller cap of 32215, which grows the set instead, kept it. For this file, every cap from
+ // the smallest that splices up must splice, and every cap returns the same six rows.
+ withTempDir { dir =>
+ val (path, _) = writeLaterSetFile(dir, lastTwoBytes = 1000)
+ val prefix = Substring(BoundReference(0, StringType, nullable = true), Literal(1), Literal(4))
+ val keys = Seq("0009", "0010", "0011", "0012", "0014", "0017").map(Literal(_))
+ def readUnder(cap: Long): (Int, Long) =
+ readUnderCap(path, laterSetSchema, Seq(In(prefix, keys)), s"${cap}b", batchSize = 4)
+ val (_, uncapped) = readUnder(64L * 1024 * 1024)
Review Comment:
Done. It is `assertSpliceCharge(24222L, rows = 6)` now, with the larger caps in a loop against the uncapped read.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3742 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{ColumnDescriptor, Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.{DataPageV1, PageReadStore}
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.filter2.columnindex.RowRanges
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat, ParquetWriter}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+import org.mockito.ArgumentMatchers.{any, anyInt}
+import org.mockito.Mockito.{doAnswer, doThrow, spy}
+import org.mockito.invocation.InvocationOnMock
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, In, IsNull, LessThan, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, Substring, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector, OnHeapColumnVector, WritableColumnVector}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false,
+ options: Map[String, String] = Map.empty): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ // Last, so a caller's options override the ones above.
+ writer.options(options).parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Asserts that a row group costs exactly `charge`: a cap of it keeps splicing, which reads the
+ // key column once, and a cap one byte under it gives splicing up, which reads it again. `read`
+ // reads the file under a cap and returns the rows it emitted with the bytes it transferred.
+ private def assertSpliceCharge(charge: Long, rows: Int)(read: String => (Int, Long)): Unit = {
+ val (uncappedRows, uncapped) = read("64MB")
+ val (atChargeRows, atCharge) = read(s"${charge}b")
+ val (underChargeRows, underCharge) = read(s"${charge - 1}b")
+ assert(Seq(uncappedRows, atChargeRows, underChargeRows) == Seq(rows, rows, rows),
+ s"every cap must keep the same $rows rows; got $uncappedRows, $atChargeRows and " +
+ underChargeRows)
+ assert(atCharge == uncapped,
+ s"a cap of exactly the charge, $charge, must keep splicing; got $atCharge against $uncapped")
+ assert(underCharge > uncapped,
+ s"a cap one byte under it must give splicing up; got $underCharge against $uncapped")
+ }
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body` as a task under `context`. The block manager has to know the task for a format
+ // reader to read its broadcast hadoop conf there.
+ private def asTask[T](context: TaskContext)(body: => T): T = {
+ val blockManager = SparkEnv.get.blockManager
+ blockManager.registerTask(context.taskAttemptId())
+ try {
+ TaskContext.withTaskContext(context)(body)
+ } finally {
+ blockManager.releaseAllLocksForTask(context.taskAttemptId())
+ }
+ }
+
+ // Runs `body`, and returns its result with how many log lines containing `text` it logged.
+ private def withLogged[T](text: String)(body: => T): (T, Int) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .count(_.contains(text))
+ (result.get, logged)
+ }
+
+ // What the reader logs when it reads a row group without page-level storage filtering.
+ private val partialReadWarning = "without page-level storage filtering"
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+ // What the reader logs when applying the filter to a row, or decoding the key pages, raises an
+ // error and it gives the filter up for that row group.
+ private val evaluationGivenUp = "applying it to a row raised an error"
+ private val decodingGivenUp = "decoding its key columns raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ onKeyReader: KeyReaderHook.OnKeyReader = null)
+ : (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap, onKeyReader = onKeyReader)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def filterOn(key: Expression): ParquetStorageFilter =
+ createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ def read(key: Expression): Unit = readAll(path, filterOn(key))._2.close()
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException =
+ TaskContext.withTaskContext(TaskContext.empty())(intercept[SparkException](read(key)))
+ // The reader does not give the filter up on a kill, so it does not report doing so.
+ val (killed, warnings) = withLogged(evaluationGivenUp) {
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ assertPropagates(killed, classOf[SparkException])
+ assert(warnings == 0, s"a kill must not be reported as the filter given up; got $warnings")
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped"))))
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in an `UnknownError`, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(internal) &&
+ internal.getCause.isInstanceOf[InternalError], s"got $internal")
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val (result, reader) =
+ readAll(path, filterOn(OnKey(bound, at = 7L, error = new AssertionError(_))))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so the fatal cause
+ // of a `NonFatal` decoding error counts down to one level less. An `InternalError` from
+ // decoding is corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ def rethrow(e: Throwable, fromRead: Boolean): Unit =
+ ParquetStorageFilter.rethrowIfMustPropagate(e, fromRead)
+ def fallsBack(e: Throwable, fromRead: Boolean): Unit =
+ try rethrow(e, fromRead) catch {
+ // ScalaTest would abort the whole suite on these rather than fail this test.
+ case t: VirtualMachineError => fail(s"the reader must fall back on $e; got $t", t)
+ }
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 1, oom), fromRead = false)) eq oom,
+ "the deepest level the executor sees counts")
+ fallsBack(under(depth, oom), fromRead = false)
+
+ assert(intercept[OutOfMemoryError](rethrow(under(depth - 2, oom), fromRead = true)) eq oom,
+ "a decoding error's chain counts one level less deep")
+ fallsBack(under(depth - 1, oom), fromRead = true)
+
+ // The depth is the executor's conf, not its default.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "2") {
+ intercept[OutOfMemoryError](rethrow(under(1, oom), fromRead = false))
+ fallsBack(under(2, oom), fromRead = false)
+ }
+ // `FileScanRDD` wraps only a `NonFatal` error, so a decoding `LinkageError` is searched as
+ // deep as the executor searches it. The OOM sits three levels down here.
+ withSparkEnvConfs(KILL_ON_FATAL_ERROR_DEPTH.key -> "3") {
+ val linkage = new ExceptionInInitializerError(new RuntimeException("static init", oom))
+ assert(intercept[OutOfMemoryError](rethrow(linkage, fromRead = true)) eq oom,
+ "a decoding error that is not NonFatal reaches the executor unwrapped")
+ // One level further down is past what the executor looks at.
+ fallsBack(new ExceptionInInitializerError(
+ new RuntimeException("static init", new RuntimeException("wraps", oom))), fromRead = true)
+ }
+
+ // A codec's `InternalError` while decoding is corrupt data, at the top or in the chain.
+ fallsBack(new InternalError("codec"), fromRead = true)
+ fallsBack(under(1, new InternalError("codec")), fromRead = true)
+ // One from an expression goes out in an `UnknownError`, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[UnknownError](rethrow(e, fromRead = false))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ }
+ fallsBack(new AssertionError("not fatal"), fromRead = false)
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM. The
+ // reader matches the exact class `reserve` throws, which a real one pins.
+ val vector = new OnHeapColumnVector(1, LongType)
+ val reserved = try intercept[RuntimeException](vector.reserve(-1)) finally vector.close()
+ assert(reserved.getClass == classOf[RuntimeException],
+ s"`reserve` must still throw the class the reader matches; got ${reserved.getClass}")
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ fallsBack(growth, fromRead = true)
+ assert(intercept[OutOfMemoryError](rethrow(growth, fromRead = false)).getMessage == "reserve")
+ // Only that exact shape. A plain exception with a fatal error deeper down is not one, and
+ // neither is a subclass with an OutOfMemoryError cause.
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new RuntimeException("wraps", under(1, oom)), fromRead = true)
+ } eq oom)
+ assert(intercept[OutOfMemoryError] {
+ rethrow(new IllegalStateException("wraps", oom), fromRead = true)
+ } eq oom)
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports. So does a
+ // vector that cannot grow. An error the executor's kill case does not take goes out as it is.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.withTaskContext(context) {
+ assert(intercept[OutOfMemoryError](rethrow(oom, fromRead = false)) eq oom)
+ // A `NonFatal` wrapper would be reported as killed, and the executor would run on.
+ assert(intercept[UnknownError] {
+ rethrow(new InternalError("expression"), fromRead = false)
+ }.getCause.isInstanceOf[InternalError])
+ val killed = intercept[SparkException](rethrow(under(1, oom), fromRead = false))
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ assert(intercept[SparkException](rethrow(growth, fromRead = true)).getCause eq growth)
+ val codec = new InternalError("codec")
+ assert(intercept[InternalError](rethrow(codec, fromRead = true)) eq codec,
+ "a plain read raises the codec's error as it is, which the executor does not report as " +
+ "killed unless ignoreCorruptFiles skips it")
+ val initError = new ExceptionInInitializerError(new IllegalStateException("static init"))
+ assert(intercept[ExceptionInInitializerError](rethrow(initError, fromRead = false)) eq
+ initError, "a LinkageError is neither NonFatal nor fatal, and goes out as it is")
+ }
+ }
+
+ test("all keys missing: a kill or a fatal error on the constant is not absorbed") {
+ // A file with none of the key columns evaluates the filter once, on the value the scan returns
+ // for them. That catch gives the filter up on the filter's own errors like the per-row one,
+ // and has to let a kill and a fatal error out like it too.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withTempDir { dir =>
+ val path =
+ writeSingleParquetFile(dir, spark.range(0, 10).selectExpr("id AS v"), 1024L)
+ val schema = StructType(Seq(
+ StructField("k", LongType).withExistenceDefaultValue("7").withCurrentDefaultValue("7"),
+ StructField("v", LongType)))
+ val bound = BoundReference(0, LongType, nullable = true)
+ def read(key: Expression): Seq[Long] = readLongs(
+ formatReader(schema, Nil, Seq(GreaterThanOrEqual(key, Literal(0L)))), path, ordinal = 1)
+
+ assert(read(bound) == (0L until 10L), "the default passes the filter, so the file is read")
+ val fatal = intercept[OutOfMemoryError] {
+ read(OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("w"))))
+ }
+ assert(fatal.getMessage == "w", s"the wrapped error itself must go out; got $fatal")
+ // As for a row, an `InternalError` here is the filter's own, not corrupt data.
+ val internal = intercept[UnknownError] {
+ read(OnKey(bound, at = 7L, error = new InternalError(_)))
+ }
+ assert(internal.getCause.isInstanceOf[InternalError], s"got $internal")
+ val killed = asTask(TaskContext.empty()) {
+ intercept[SparkException] {
+ read(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_)))
+ }
+ }
+ assert(killed.getMessage.contains("task kill") &&
+ !DataSourceUtils.shouldIgnoreCorruptFileException(killed), s"got $killed")
+ }
+ }
+ }
+
+ test("a give-up WARN is reported once per split, and a give-up stays in its row group") {
+ // The first and the last of three row groups give the filter up on an evaluation error, and the
+ // reader reports that once. The middle one raises nothing, so the filter must still drop its
+ // keys 151 to 200, which a reader that gave the filter up for the rest of the split would not.
+ withTempDir { dir =>
+ val rows = (1L to 300L).map(i => (i, s"v_$i"))
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "100"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(100L, 100L, 100L),
+ "the fixture must be three row groups of 100 rows")
+ val bound = BoundReference(0, LongType, nullable = true)
+ def failingAt(key: Long): Expression = GreaterThanOrEqual(
+ OnKey(bound, at = key, error = new IllegalStateException(_)), Literal(0L))
+ val dropsMiddle = Or(LessThanOrEqual(bound, Literal(150L)), LessThan(Literal(200L), bound))
+ val filter = createFilter(Seq(And(And(failingAt(5L), failingAt(250L)), dropsMiddle)),
+ kvSchema())
+ val ((result, reader), warnings) = withLogged(evaluationGivenUp)(readAll(path, filter))
+ try {
+ assert(result == rows.filter { case (k, _) => k <= 150L || k > 200L },
+ s"the middle row group must be filtered and the other two come back whole; got " +
+ s"${result.size} rows")
+ assert(warnings == 1, s"the evaluation WARN must be reported once; got $warnings")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ val thrown = TaskContext.withTaskContext(TaskContext.empty()) {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // The rows the scans with storage filters excluded, by both row metrics. Through `collect(plan)`,
+ // which finds those scans in a plan AQE wrapped too.
+ private def rowsExcludedByStorageFilters(plan: SparkPlan): Long =
+ collect(plan) {
+ case s: FileSourceScanExec if s.storageFilters.nonEmpty =>
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ s.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ }.sum
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val excluded = rowsExcludedByStorageFilters(plan)
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ assertSpliceCharge(103L, rows = 7) { cap =>
+ val (rows, bytes, _) = run(cap, 394L)
+ (rows, bytes)
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // The later-set tests' file: twenty string keys, `0001` to `0020` padded with `x` to 1000 bytes,
+ // the last two to `lastTwoBytes`, in one row group. Returns it with the rows `laterSetFilter`
+ // keeps.
+ private def writeLaterSetFile(dir: File, lastTwoBytes: Int): (String, Seq[(String, String)]) = {
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), " +
+ s"REPEAT('x', IF(id >= 19, ${lastTwoBytes - 4}, 996))) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val kept = (15 to 20).map { id =>
+ (f"$id%04d" + "x" * ((if (id >= 19) lastTwoBytes else 1000) - 4), id.toString)
+ }
+ (path, kept)
+ }
+
+ private val laterSetSchema: StructType = kvSchema(keyType = StringType)
+
+ // `k >= "0015"`, which keeps the last six keys. Over a capacity of 4 that is one set of four,
+ // then two more.
+ private val laterSetFilter: Seq[Expression] = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ // Reads a later-set file over a capacity of 4 under `cap`. Returns the rows, and the byte-child
+ // capacities the second set's rows came back in.
+ private def readLaterSet(path: String, cap: Long): (Seq[(String, String)], Seq[Int]) = {
+ val (rows, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(laterSetFilter, laterSetSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString,
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ (rows.map(r => (r._1, r._2)), rows.drop(4).map(_._3).distinct)
+ }
+
+ test("a later accumulator set is sized for the survivors it can expect and charged once") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, scaled to the survivors it can still expect at the rate seen since the row group's
+ // first survivor. That memory is taken before any value arrives, so it is charged then. Six
+ // survivors over a capacity of 4 fill one set of four 1000-byte keys, and the second set starts
+ // two rows before the end of the row group, with every row since the first survivor kept. So
+ // it is sized for two of those values, 2000 bytes, charged 8000. Its own two keys are 2000
+ // bytes each, so they add 8000 more once the sizing is used up, and the row group holds 32142
+ // bytes in all. A cap of 32142 keeps splicing and one of 32141 gives it up, which a missing or
+ // a double charge would each get wrong. The second set's byte child is reserved for 2000
+ // bytes, which `reserve` doubles to 4000, where sizing it as a full set would have taken 8000.
+ // And the keys must come back right whichever path the row group took.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 2000)
+ assertSpliceCharge(32142L, rows = 6) { cap =>
+ readUnderCap(path, laterSetSchema, laterSetFilter, cap, batchSize = 4)
+ }
+
+ Seq(Long.MaxValue, 32142L, 32141L).foreach { cap =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected,
+ s"every survivor must come back with its own key under cap $cap; got ${rows.map(_._2)}")
+ if (cap >= 32142L) {
+ assert(secondSet == Seq(4000),
+ s"the second set must be sized for the two values left under cap $cap; got $secondSet")
+ }
+ }
+ }
+ }
+
+ test("a later accumulator set is sized only when it fits with its rows' fixed part and ranges") {
+ // The second set's two keys are 500 bytes here, shorter than the first set's 1000, so growing
+ // value by value is the cheaper path: 20142 bytes in all against 24142 sized. A cap of 24222
+ // sizes the set, whose byte child is then reserved for 2000 bytes and holds 4000. One of
+ // 24221 leaves 8113 bytes after the first set and its range, a byte short of the 8000 of
+ // sizing, the 34 of the two rows' fixed part and the 80 of a range for each. So the set grows
+ // instead, to a 1000-byte child, and keeps splicing. A cap of 20141 gives splicing up.
+ withTempDir { dir =>
+ val (path, expected) = writeLaterSetFile(dir, lastTwoBytes = 500)
+ def run(maxSplicedBytes: String): Long =
+ readUnderCap(path, laterSetSchema, laterSetFilter, maxSplicedBytes, batchSize = 4)._2
+
+ val uncapped = run("64MB")
+ assert(run("24222b") == uncapped, "a cap that fits the sizing must keep splicing")
+ assert(run("24221b") == uncapped, "and so must one that only fits growing value by value")
+ assert(run("20141b") > uncapped, "a cap under both must give splicing up")
+
+ Seq(Long.MaxValue -> 4000, 24222L -> 4000, 24221L -> 1000).foreach { case (cap, child) =>
+ val (rows, secondSet) = readLaterSet(path, cap)
+ assert(rows == expected, s"every survivor must come back with its own key under cap $cap")
+ assert(secondSet == Seq(child),
+ s"under cap $cap the second set's byte child must hold $child bytes; got $secondSet")
+ }
+ }
+ }
+
+ test("a later set is sized only when the ranges of its scattered survivors fit too") {
+ // Keys 9 to 12 fill the first set over a capacity of 4, and keys 14 and 17 come scattered, each
+ // in a range of its own. The second set opens at key 14, sized for the four rows the rate
+ // since the first survivor promises, and its sizing is charged up front. So it is sized only
+ // when the cap holds a range for each of those rows as well. Without that, caps of 32216 to
+ // 32221 sized the set and then passed the cap on key 17's range, giving splicing up where a
+ // smaller cap of 32215, which grows the set instead, kept it. For this file, every cap from
+ // the smallest that splices up must splice, and every cap returns the same six rows.
+ withTempDir { dir =>
+ val (path, _) = writeLaterSetFile(dir, lastTwoBytes = 1000)
+ val prefix = Substring(BoundReference(0, StringType, nullable = true), Literal(1), Literal(4))
+ val keys = Seq("0009", "0010", "0011", "0012", "0014", "0017").map(Literal(_))
+ def readUnder(cap: Long): (Int, Long) =
+ readUnderCap(path, laterSetSchema, Seq(In(prefix, keys)), s"${cap}b", batchSize = 4)
+ val (_, uncapped) = readUnder(64L * 1024 * 1024)
+ val (smallerRows, smaller) = readUnder(24221L)
+ assert(smallerRows == 6 && smaller > uncapped, "the smallest cap that splices must be 24222")
+ Seq(24222L, 32215L, 32216L, 32221L, 32222L).foreach { cap =>
+ val (rows, bytes) = readUnder(cap)
+ assert(rows == 6 && bytes == uncapped,
+ s"a cap of $cap must keep the six rows and splicing; got $rows rows")
+ }
+ }
+ }
+
+ test("a sparse last set is sized for the survivors it can expect") {
+ // Every tenth of 125 keys survives, so over a capacity of 4 the fourth set holds key 120 alone.
+ // It opens five rows before the end of the row group, at a rate of 13 survivors in 121 rows, so
+ // it expects one survivor and is sized for one 1000-byte value, which `reserve` doubles to a
+ // 2000-byte child. Sized for the five rows left, as a full set, it would have taken 8000. The
+ // two sets before it are sized as full ones, since the rate promises them four survivors each.
+ withTempDir { dir =>
+ val df = spark.range(0, 125).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), REPEAT('x', 996)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 1024L * 1024)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val lastDigit =
+ Substring(BoundReference(0, StringType, nullable = true), Literal(4), Literal(1))
+ val (rows, reader) = readAllWith(path, Seq("k", "v"),
+ createFilter(Seq(EqualTo(lastDigit, Literal("0"))), laterSetSchema),
+ (b, i) => (b.column(0).getUTF8String(i).toString.take(4),
+ ParquetTestAccess.byteChildCapacity(b.column(0).asInstanceOf[WritableColumnVector])),
+ capacity = 4)
+ reader.close()
+ assert(rows.map(_._1) == (0 to 120 by 10).map(id => f"$id%04d"), s"got ${rows.map(_._1)}")
+ val children = rows.map(_._2)
+ assert(children.slice(4, 12).distinct == Seq(8000) && children.drop(12) == Seq(2000),
+ s"the sparse last set must be sized for its one survivor; got $children")
+ }
+ }
+
+ test("two key columns are both charged") {
+ // Seven survivors in one range cost 7 * (9 + 9) + 40 = 166 bytes with two long keys. A cap of
+ // 166 keeps splicing and one of 165 gives it up, so charging one key alone would show.
+ withTempDir { dir =>
+ val df = spark.range(1, 401).selectExpr("id AS k", "id AS k2", "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val schema = StructType(Seq(StructField("k", LongType), StructField("k2", LongType),
+ StructField("v", StringType)))
+ val storageFilters = Seq(And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(394L)),
+ GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(0L))))
+ assertSpliceCharge(166L, rows = 7) { cap =>
+ readUnderCap(path, schema, storageFilters, cap, batchSize = 16)
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warnings) = withLogged(partialReadWarning) {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(warnings == 0,
+ s"the footer check must get there before the exception path; got $warnings")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warnings) = withLogged(partialReadWarning) {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(warnings == 0, "the footer check must decline the row group rather than the read " +
+ s"discovering it; got $warnings")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ val path = writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize = 64L * 1024 * 1024,
+ options = Map(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT -> "100"))
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ // The rows a reader hands back before it throws, what it throws, and, for the filtering
+ // reader, whether its scratch vectors were replaced on the way and the old ones freed. Both
+ // read off heap, where an old scratch vector left open would leak native memory.
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable, Boolean) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(true, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(true, 16, storageFilter)
+ }
+ def scratch: Seq[WritableColumnVector] = reader match {
+ case filtering: LateMaterializationParquetRecordReader =>
+ ParquetTestAccess.keyScratchVectors(filtering)
+ case _ => Nil
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val scratchBefore = scratch
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ val replacedAndFreed = scratch.zip(scratchBefore).exists { case (now, before) =>
+ (now ne before) && before.asInstanceOf[OffHeapColumnVector].valuesNativeAddress() == 0
Review Comment:
Done. One suite-level `address`, noted as fixed-width only, serves all four checks, and the hook looks the scratch vectors up once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-6033916584
@cloud-fan thank you again for your review. If you still have an open question on any of your threads, I'm happy to address it in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5820024251
Thank you for the second round. It was as thorough as the first, and the individual replies carry the details.
The comment that mattered most was the design question on treating the extracted filter as an optional hint. That is what the PR does now, and it went one step further than the suggestion: the conjunct is not moved out of the plan at all, it is copied. It stays in the post-scan `Filter`, so the reader may give a file or a row group up freely, and `UnsupportedFileReadException`, its case in the shared classifier and the hard failure on a page-index-less file are all gone.
Two things settled that direction. The bloom is already built by the time the scan runs, so having the `Filter` use it regardless costs nothing extra and means it is never built for nothing. And whoever turns this conf on is doing it because the filter prunes a lot, so evaluating it a second time on the rows that survive is a small price rather than a regression.
The description is rewritten around that, with a Design decisions section. It also now carries a measurement of splicing against re-reading the key columns, which answers your earlier question with numbers: re-reading transfers up to 49% more bytes than splicing, and more than the feature being off, because phase 1 has already read the key column once.
Everything is in c255f88.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4098089794
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -409,23 +622,244 @@ public boolean nextBatch() throws IOException {
}
cv.assemble();
}
- // If needed, compute row indexes within a file.
- if (rowIndexGenerator != null) {
- rowIndexGenerator.populateRowIndex(columnVectors, num);
- }
+ }
- rowsReturned += num;
+ /** Publishes {@code num} rows as the current batch. */
+ private void finishBatch(int num) {
columnarBatch.setNumRows(num);
+ rowsReturned += num;
numBatched = num;
batchIdx = 0;
- return true;
+ }
+
+ /**
+ * Points the batch's key slots at one survivor key vector each, which is what publishes them: the
+ * same {@link ColumnarBatch} is handed out every time, over an array it holds by reference. The
+ * queues keep owning those vectors until the next batch releases them, so a phase-2 read that
+ * throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void publishSurvivorKeyVectors(int num) {
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ if (keyVectorQueues[i].isEmpty()) {
+ // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop
+ // is driven by that same count. Named rather than left to NoSuchElementException.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to "
+ + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num));
+ }
+ }
+ keyVectorsPublished = true;
+ // Key slots are filled in ascending slot order while `keyIdx` walks the queues in key-list
+ // order, so the pairing is the identity only because `ParquetStorageFilter.create` sorts
+ // `keyColumnIndices` ascending. `isKeyTopLevel` says which slots are keys, not where each
+ // sits in that list, so this loop cannot re-derive the pairing: an unsorted list would swap
+ // key columns in the output batch. Only key slots are touched, since a non-key slot never
+ // holds anything but its persistent vector.
+ int keyIdx = 0;
+ for (int i = 0; i < isKeyTopLevel.length; i++) {
+ if (isKeyTopLevel[i]) spliceBatchColumns[i] = keyVectorQueues[keyIdx++].peekFirst();
+ }
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (java.util.ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
missingColumns = new HashSet<>();
for (ParquetColumn column : CollectionConverters.asJava(parquetColumn.children())) {
checkColumn(column);
}
+ if (storageFilter != null) {
+ initializeLateMaterialization();
+ }
+ }
+
+ /**
+ * Sets the storage filter for late materialization. Must be called before {@link #initialize};
+ * {@link #initializeLateMaterialization()} then inspects the per-file schema and decides whether
+ * splicing engages.
+ *
+ * <p>It does not engage in one real case: every key column is missing from this physical file
+ * under schema evolution. The predicate is then rewritten with each missing key replaced by the
+ * constant the reader materializes for it (its existence DEFAULT, else null) and evaluated once.
+ * True keeps the file unfiltered, false or null skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by
+ * {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}, so a
+ * violation of it here is a planner bug and is asserted rather than handled. A file the reader
+ * simply cannot prune is a different matter: it reads it the way a plain scan would.
+ */
+ public void setStorageFilter(ParquetStorageFilter storageFilter) {
+ this.storageFilter = storageFilter;
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ lateMatReader = reader.getUnderlyingReader();
+ if (lateMatReader == null) {
+ // Late materialization drives a ParquetFileReader directly, so without one the filter is
+ // simply not applied and the plain read path runs. The conjunct is in the post-scan filter as
+ // well, so the rows it would have dropped are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ useColumnIndexFilter = configuration.getBoolean(
+ ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn. Partition into present and missing (the
+ // latter can happen under schema evolution: a column is in the requested schema but not in this
+ // physical parquet file). For any non-primitive key we still bail; phase-1 reads only primitive
+ // leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> missingKeyLocalPositions = new ArrayList<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (missingColumns.contains(column)) {
+ missingKeyLocalPositions.add(i);
+ } else {
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // If any key column is missing from this file, rewrite the predicate to substitute the constant
+ // the reader will actually materialize for that column. That is the column's existence DEFAULT
+ // when it has one (ParquetColumnVector writes it into the output vector and marks the vector
+ // constant), otherwise null. Substituting null for a column that reads back as its default
+ // would filter on a value the scan never returns. The predicate must be evaluated against the
+ // substituted constant rather than skipped: null does not always mean false in a filter.
+ if (!missingKeyLocalPositions.isEmpty()) {
+ int[] missing = new int[missingKeyLocalPositions.size()];
+ Object[] missingValues = new Object[missingKeyLocalPositions.size()];
+ Object[] existenceDefaults =
+ ResolveDefaultColumns.existenceDefaultValues(sparkRequestedSchema);
+ for (int i = 0; i < missing.length; i++) {
+ missing[i] = missingKeyLocalPositions.get(i);
+ missingValues[i] = existenceDefaults[keyIndices[missing[i]]];
+ }
+ storageFilter = storageFilter.rewriteForMissingKeys(missing, missingValues);
+
+ if (presentKeyColumns.isEmpty()) {
+ // Every key column is missing, so the rewritten predicate is constant for this file.
+ boolean keepAll = storageFilter.evalAllMissing();
+ if (!keepAll) recordFileSkipped();
+ storageFilter = null;
+ hitEndOfData = !keepAll;
+ return;
+ }
+ }
+
+ keyDescriptors = new ColumnDescriptor[presentKeyColumns.size()];
+ keyRequired = new boolean[presentKeyColumns.size()];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < presentKeyColumns.size(); i++) {
+ ParquetColumn column = presentKeyColumns.get(i);
+ keyDescriptors[i] = column.descriptor().get();
+ keyRequired[i] = column.required();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ requestedColumns = requestedSchema.getColumns();
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. A null
+ // `nonKeyColumns` says there is nothing for phase 2 to read, so it is skipped entirely. The
+ // planner does not push a scan of that shape, since such a scan reads what a plain one would,
+ // so this is reachable only by driving the reader directly.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount > 0) {
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ }
+ initializeSplicingState(presentKeyColumns);
+
+ totalBlockCount = lateMatReader.getRowGroups().size();
+ nextBlockIndex = 0;
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues. {@link #initBatch} also reads it, as the sign that this file splices at
+ * all and needs its own batch array.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState(List<ParquetColumn> presentKeyColumns) {
+ int numTop = sparkSchema.fields().length;
+ isKeyTopLevel = new boolean[numTop];
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int slot : keyIndices) {
+ isKeyTopLevel[slot] = true;
+ }
+ int numKeys = presentKeyColumns.size();
+ keyVectorQueues = new java.util.ArrayDeque[numKeys];
+ for (int i = 0; i < numKeys; i++) {
+ keyVectorQueues[i] = new java.util.ArrayDeque<>();
+ }
+ currentKeyAccumulators = new WritableColumnVector[numKeys];
+ currentKeyAccumulatorRowCount = 0;
+ keyCopiers = new ValueCopier[numKeys];
+ keyVariableLength = new boolean[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ DataType dt = fields[keyIndices[i]].dataType();
+ keyCopiers[i] = copierFor(dt);
+ keyVariableLength[i] = isVariableLength(dt);
+ // One null byte per row either way. A fixed-width value adds its own width, a variable-length
+ // one the int offset and int length that point at the byte child.
+ keyFixedBytesPerRow += keyVariableLength[i] ? 1 + 8 : 1 + dt.defaultSize();
+ }
+ }
+
+ /**
+ * What phase 2 of the current row group will hold for its row ranges. Every column reader it
+ * drives materializes the row group's range list of its own ({@code ParquetReadState}), and a
+ * filter whose survivors are scattered makes one range per surviving row.
+ */
+ private long rowRangeStateBytes(long rangeCount) {
Review Comment:
This multiplies the range budget by the number of phase-2 leaf columns because `ParquetReadState.constructRanges` builds its own copy of the range list for every column reader. For a wide projection, the filter is given up at a small number of ranges: 64MB / (40B x 100 leaves) is about 16.8k ranges, which a 1M-row row group easily reaches with scattered survivors (e.g., the bloom filter's false positives).
The give-up happens only after phase 1 has already read, decoded, and evaluated the key columns, and then phase 2 re-reads the whole projection including the keys. The decision is not memoized per file (unlike `fileHasNoOffsetIndex`), so this repeats for every row group, which is slower than having the feature off. The conf doc's "as slow as not pushing the filter" understates this.
Since Parquet 1.18.1 exposes `RowRanges.getRanges()` and public `Range.from`/`to`, could `ParquetReadState` iterate one shared, immutable range list instead of copying it per column? That removes the `x leaves` factor at the root.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Not applying the storage filter to {}: reading part of a row group needs a "
+ + "Parquet offset index, and this file was written without a page index for at least "
+ + "one projected column", e, MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is computed for a row group whose filter was given up: it read what a plain scan
+ // reads, so the answer is a certain zero. `needBytes`, not just `bytesAvoidedPf != null`,
+ // because that is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null && !filterGivenUp) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the baseline
+ // after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. `getRowRanges` is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then. It has to be
+ // guarded the same way phase 0 guards it, since it consults the pushed filter but not the
+ // conf that turns column-index filtering off.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
+ // The ranges being built are about to be thrown away, so stop evaluating the rest.
+ if (filterGivenUp) return null;
+ }
+ }
+ }
+ remaining -= num;
+ }
+
+ if (accumulate) {
+ finalizePartialAccumulators();
+ }
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ // Assigned before the loop on purpose: an allocation failure part way through then leaves the
+ // vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows
+ * each.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, pushing them onto their
+ * queues once full. All key columns advance in lockstep, which is what keeps the queues aligned.
+ *
+ * <p>Returns false when the buffered survivors have passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} and this row group has
+ * given splicing up, in which case it has already released what it held and the caller must stop
+ * calling this.
+ */
+ private boolean appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ long valueBytes = 0L;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ // Measured on the destination: a dictionary-encoded source has no length of its own, since
+ // its values are read through the dictionary.
+ if (keyVariableLength[i]) valueBytes += dst.getArrayLength(dstRow);
+ }
+ }
+ splicedBytes += keyFixedBytesPerRow + valueBytes;
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) {
Review Comment:
`splicedBytes` is updated per row, but it is compared against `maxSplicedRowGroupBytes` only here, when an accumulator reaches `capacity` rows. The post-phase-1 check (around line 1023) only looks at `rowRangeStateBytes(survivorRangeCount)`, not at `splicedBytes`.
As a result, the advertised bound is not enforced:
- A row group whose survivors fit in a single accumulator is never weighed. With 32KB string keys, 4095 survivors buffer about 128MB against the 64MB cap, and the buffer is kept through phase 2 and emit.
- When survivors span multiple batches, the first check can already be one full accumulator past the cap.
In addition, the conf doc says such a row group "holds no more than the plain read path does", but during phase 1 the reader holds both the decoded capacity-sized `keyScratchVectors` (all rows) and the survivor accumulators, i.e., about twice the plain path's key memory, outside any `MemoryConsumer`. Could we check the cap per row (it is a single comparison) and include `splicedBytes` in the post-loop check, and update the doc?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
Review Comment:
Phase 0 honors `parquet.filter.columnindex.enabled=false`, which is the documented escape hatch for a file with a wrong page index. However, this `readFilteredRowGroup(blockIdx, finalRanges)` still selects pages and synchronizes rows through the offset index whenever `finalRanges` is a strict subset of the block. In Parquet 1.18.1, `readFilteredRowGroup(int, RowRanges)` does not check `options.useColumnIndexFilter()`, unlike `readNextFilteredRowGroup()`, and `VectorizedColumnReader` takes `pageFirstRowIndex` from `page.getFirstRowIndex()`.
So, for a file with a corrupt offset index, the plain path reads it correctly (with the escape hatch on, or with no pushed data filter at all, where it never touches the offset index). With this feature on, the phase-1 keys are still correct, but the phase-2 non-key values can come from the wrong rows. Since the post-scan `Filter` only re-checks the key hash, this becomes a silent wrong result. Could we skip late materialization (or read whole row groups in phase 2) when `useColumnIndexFilter` is false?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -409,23 +622,244 @@ public boolean nextBatch() throws IOException {
}
cv.assemble();
}
- // If needed, compute row indexes within a file.
- if (rowIndexGenerator != null) {
- rowIndexGenerator.populateRowIndex(columnVectors, num);
- }
+ }
- rowsReturned += num;
+ /** Publishes {@code num} rows as the current batch. */
+ private void finishBatch(int num) {
columnarBatch.setNumRows(num);
+ rowsReturned += num;
numBatched = num;
batchIdx = 0;
- return true;
+ }
+
+ /**
+ * Points the batch's key slots at one survivor key vector each, which is what publishes them: the
+ * same {@link ColumnarBatch} is handed out every time, over an array it holds by reference. The
+ * queues keep owning those vectors until the next batch releases them, so a phase-2 read that
+ * throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void publishSurvivorKeyVectors(int num) {
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ if (keyVectorQueues[i].isEmpty()) {
+ // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop
+ // is driven by that same count. Named rather than left to NoSuchElementException.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to "
+ + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num));
+ }
+ }
+ keyVectorsPublished = true;
+ // Key slots are filled in ascending slot order while `keyIdx` walks the queues in key-list
+ // order, so the pairing is the identity only because `ParquetStorageFilter.create` sorts
+ // `keyColumnIndices` ascending. `isKeyTopLevel` says which slots are keys, not where each
+ // sits in that list, so this loop cannot re-derive the pairing: an unsorted list would swap
+ // key columns in the output batch. Only key slots are touched, since a non-key slot never
+ // holds anything but its persistent vector.
+ int keyIdx = 0;
+ for (int i = 0; i < isKeyTopLevel.length; i++) {
+ if (isKeyTopLevel[i]) spliceBatchColumns[i] = keyVectorQueues[keyIdx++].peekFirst();
Review Comment:
This pairs the k-th key slot (in ascending slot order) with queue k, which is correct only because `ParquetStorageFilter.create` sorts the ordinals. The comments document it, but nothing enforces it, while the rest of the reader pairs queue `i` with `keyColumnIndices[i]` directly. If a future change breaks the ascending order, key columns would be swapped in the output without any error. How about `spliceBatchColumns[keyIndices[k]] = keyVectorQueues[k].peekFirst()` to remove the dependency?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Not applying the storage filter to {}: reading part of a row group needs a "
+ + "Parquet offset index, and this file was written without a page index for at least "
+ + "one projected column", e, MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is computed for a row group whose filter was given up: it read what a plain scan
+ // reads, so the answer is a certain zero. `needBytes`, not just `bytesAvoidedPf != null`,
+ // because that is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null && !filterGivenUp) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the baseline
+ // after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. `getRowRanges` is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then. It has to be
+ // guarded the same way phase 0 guards it, since it consults the pushed filter but not the
+ // conf that turns column-index filtering off.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
Review Comment:
The correctness here relies on an implicit ordering invariant: Parquet caches the `ColumnIndexStore` per block with the `paths` at the time of the first call, and `setRequestedSchema` does not invalidate it. For a path outside the cached set, `MISSING_INDEX_STORE.getOffsetIndex()` returns `null` rather than throwing. I verified that the current phase ordering never builds the store with a narrower path set than a later lookup, but a future change that calls this before phase 2 (e.g., to estimate bytes for gating) would make the subsequent read NPE inside Parquet's `filterOffsetIndex(null, ...)` (not the `MissingOffsetIndexException` this PR catches) and would silently undercount the metrics here. Could we snapshot the per-block ranges/stores with the full projection at initialization, so the invariant is enforced by code rather than by comments?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:
##########
@@ -230,6 +274,34 @@ class ParquetFileFormat
val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead
val archiveFormatEnabled = parquetOptions.archiveFormatEnabled
+ // Late materialization needs the vectorized reader. `enableVectorizedReader` is recomputed from
+ // the live session conf when the RDD is built, so a flip of
+ // spark.sql.parquet.enableVectorizedReader (or the nested-column variant) after planning lands
+ // here, and the filters are simply not installed: the post-scan Filter still holds them.
+ val storageFilterOpt: Option[ParquetStorageFilter] = if (storageFilters.isEmpty) {
+ None
+ } else if (!enableVectorizedReader) {
+ logInfo(log"Not honoring storage filters for schema " +
+ log"${MDC(SCHEMA, resultSchema.catalogString)}: the vectorized Parquet reader is disabled")
+ None
+ } else {
+ val metrics = StorageFilterMetrics(
+ rowGroupsSkipped = storageFilterMetrics.getOrElse(
+ FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED, null),
+ rowsExcludedByRowGroup = storageFilterMetrics.getOrElse(
+ FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP, null),
+ rowsExcludedWithinRowGroup = storageFilterMetrics.getOrElse(
+ FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP, null),
+ bytesAvoidedByRowGroup = storageFilterMetrics.getOrElse(
+ FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP, null),
+ bytesAvoidedByPageFiltering = storageFilterMetrics.getOrElse(
+ FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING, null))
+ // `create` requires every condition extractStorageFilters already pre-checked, so a violation
Review Comment:
nit. `extractStorageFilters` doesn't exist. It seems to have been renamed to `FileSourceStrategy.storageFiltersFor`. The same stale reference is in `ParquetStorageFilter.scala` (line 150) and `ParquetStorageFilterSuite.scala` (line 339). In addition, the test comments at lines 285 (`pendingCloseKeyVectors`) and 1416 ("ColumnarBatch is replaced on every nextBatch()") describe an older design, while the current reader reuses one batch object and rewrites its key slots in place.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayOutputStream, File}
+import java.net.URI
+import java.time.LocalTime
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapreduce.Job
+import org.apache.parquet.column.Encoding
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetInputFormat, ParquetOutputFormat}
+
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, XxHash64}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FileSourceScanLike, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests the late-materialization path of [[VectorizedParquetRecordReader]] driven by a
+ * [[ParquetStorageFilter]]. Writes small multi-row-group parquet files, wires a hand-built filter
+ * into the reader, and asserts correctness + the two storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false), StructField("v", StringType, nullable = false)))
+ ParquetStorageFilter.create(Seq(expr), requested, metrics)
+ }
+
+ // Writes a single-column (just `k`) parquet file for the supplied key type via Spark's
+ // {@code Encoder}.
+ private def writeKeyOnlyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(dir, spark.createDataset(keys).toDF("k"), rowGroupSize)
+
+ // Reads a key-only file, returning the survivor keys and the reader. The {@code extract} function
+ // pulls one value at a time from the batch's key column.
+ private def readKeyOnlyAll[T](
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T,
+ capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k"), storageFilter, (batch, i) => extract(batch.column(0), i),
+ capacity)
+
+ // Builds a `k >= threshold` storage filter bound to position 0 against a key-only schema of the
+ // given key type.
+ private def keyOnlyAtLeastFilter(
+ threshold: Literal,
+ keyType: DataType,
+ metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, keyType, nullable = false), threshold)
+ val requested = StructType(Seq(StructField("k", keyType, nullable = false)))
+ ParquetStorageFilter.create(Seq(expr), requested, metrics)
+ }
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows in one row group: parquet's first row-group size check is at record 100, so
+ // `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = keyAtLeastFilter(1000L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf,
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(rgSkipped.value > 0,
+ s"expected at least one row group skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedRg.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${rowsExcludedRg.value}")
+ assert(rowsExcludedPf.value == 0,
+ s"no partial-row-group filtering expected; got ${rowsExcludedPf.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(bytesAvoidedRg.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got ${bytesAvoidedRg.value}")
+ assert(bytesAvoidedPf.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got ${bytesAvoidedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val filter = keyAtLeastFilter(0L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped, rowsExcludedWithinRowGroup = rowsExcludedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(rgSkipped.value == 0, s"nothing should be skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedPf.value == 0,
+ s"nothing should be filtered; got ${rowsExcludedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("mixed: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ // Output is exact: VectorizedColumnReader uses PageReadStore.getRowIndexes (driven by
+ // our finalRanges) to skip rows within partial pages, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it: a row group that was neither skipped whole nor emitted has
+ // to have its rows counted there.
+ assert(rowsExcludedPf.value > 0,
+ s"expected rows excluded inside a kept row group; got ${rowsExcludedPf.value}")
+ assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == rows.size,
+ s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " +
+ s"should account for all ${rows.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key-only projection: phase 2 is skipped and both byte-avoided metrics are zero") {
+ // When the projected schema contains only the bloom key, phase 2 is skipped entirely
+ // (`nonKeyColumns == null` in the reader). All output rows come from the per-key-column
+ // queues populated in phase 1. Total bytes read match the no-storage-filter path (phase 1 reads
+ // the key column once instead of phase 2 re-reading it), so both `avoided` metrics are zero:
+ // there are no non-key bytes to skip. This is the shape the design notes call the biggest win,
+ // so it must not be the shape that pays for metrics.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ // k >= 195 -> last 6 keys survive; preceding row groups skipped or page-pruned.
+ val filter = keyOnlyAtLeastFilter(Literal(195L), LongType, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf,
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i))
+ try {
+ val expected = keys.filter(_ >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact survivor keys; got ${result.sorted}, expected ${expected.toSeq.sorted}")
+ assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedRg.value > 0,
+ s"a skipped row group must count its rows too; got ${rowsExcludedRg.value}")
+ // The all-keys path takes its kept-row count from `finalRowCount` rather than from a
+ // phase-2 page store, so this identity is the only thing that checks that arithmetic.
+ assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == keys.size,
+ s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " +
+ s"should account for all ${keys.size} rows")
+ // For an all-keys projection, the no-storage-filter path would have read the same key
+ // column phase 1 reads. There are no non-key bytes to avoid; both metrics are 0.
+ assert(bytesAvoidedRg.value == 0,
+ s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedRg.value}")
+ assert(bytesAvoidedPf.value == 0,
+ s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("multi-batch emit: survivor count exceeds capacity") {
+ // Drive the reader at capacity = 16 with a row group of 100 surviving rows. Exercises:
+ // - The per-key-column queue holding multiple full-capacity vectors plus a partial tail.
+ // - pendingCloseKeyVectors getting closed at the start of every subsequent emit.
+ // - The per-emit ColumnarBatch reconstruction running ceil(100/16) = 7 times.
+ withTempDir { dir =>
+ val keys = (1L to 100L)
+ // Big rowGroupSize so all 100 rows fit in one row group.
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 64 * 1024L)
+ // Filter accepts every row so the queue is fully populated.
+ val filter = keyOnlyAtLeastFilter(Literal(0L), LongType)
+ val (result, reader) =
+ readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i), capacity = 16)
+ try {
+ assert(result == keys.toSeq,
+ s"expected all keys returned in order across multiple batches; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("int key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the IntegerType branch of ValueCopier.
+ withTempDir { dir =>
+ val keys = (1 to 100)
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyOnlyAtLeastFilter(Literal(90), IntegerType)
+ val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getInt(i))
+ try {
+ assert(result.toSet == keys.filter(_ >= 90).toSet,
+ s"expected int keys >= 90; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("string key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the StringType branch of ValueCopier (variable-length byte copy via putByteArray).
+ withTempDir { dir =>
+ val keys = (1 to 20).map(i => f"k$i%03d")
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyOnlyAtLeastFilter(
+ Literal.create("k015", StringType), StringType)
+ val (result, reader) =
+ readKeyOnlyAll(path, filter, (vec, i) => vec.getUTF8String(i).toString)
+ try {
+ assert(result.toSet == keys.filter(_ >= "k015").toSet,
+ s"expected string keys >= 'k015'; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction: extractStorageFilters pre-checks each one, and
+ // by the time create runs the conjunct is gone from the post-scan Filter, so a soft rejection
+ // would silently return rows the filter excludes. create fails instead.
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+
+ // Nothing to push: the caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+ }
+
+ // Serializes a [[BloomFilter]] to bytes suitable for a [[Literal]].
+ private def bloomBytes(bf: BloomFilter): Array[Byte] = {
+ val out = new ByteArrayOutputStream()
+ bf.writeTo(out)
+ out.toByteArray
+ }
+
+ // Computes `XxHash64(v)` using the same seed Spark uses for runtime bloom filters.
+ private def xxHash64(v: Any, dt: DataType): Long = {
+ new XxHash64(Seq(Literal(v, dt))).eval(InternalRow.empty).asInstanceOf[Long]
+ }
+
+ test("rewriteForMissingKeys: regular equi-join bloom probes the null key's hash") {
+ // Regular equi-join: the runtime bloom is BloomFilterMightContain(bloom, XxHash64(key)).
+ // XxHash64 is a HashExpression, so it is nullable = false and hashes a null input to its SEED
+ // rather than producing null. Substituting Literal(null) therefore leaves a concrete probe for
+ // `xxHash64(null)`, and whether the file is kept depends on whether that hash is in the bloom.
+ // Both directions are asserted so the outcome does not hinge on a lucky bloom miss.
+ val nullKeyHash = xxHash64(null, LongType)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+
+ def rewriteWithBloomContaining(hashes: Long*): ParquetStorageFilter = {
+ val bf = BloomFilter.create(10, 128)
+ hashes.foreach(bf.putLong)
+ val expr = BloomFilterMightContain(
+ Literal(bloomBytes(bf), BinaryType),
+ new XxHash64(Seq(BoundReference(0, LongType, nullable = true))))
+ val filter = ParquetStorageFilter.create(Seq(expr), requested)
+ filter.rewriteForMissingKeys(Array(0), Array(null))
+ }
+
+ val dropped = rewriteWithBloomContaining(xxHash64(42L, LongType))
+ assert(dropped.keyColumnIndices.isEmpty,
+ "all key positions are missing; keyColumnIndices should be empty")
+ assert(!dropped.evalAllMissing(),
+ "the null key's hash is not in the bloom, so evalAllMissing must return false " +
+ "(skip the file)")
+
+ val kept = rewriteWithBloomContaining(nullKeyHash)
+ assert(kept.evalAllMissing(),
+ "the null key's hash IS in the bloom, so evalAllMissing must return true (keep the file)")
+ }
+
+ test("rewriteForMissingKeys: missing key with an existence DEFAULT probes the default's hash") {
+ // A missing column that has a non-null existence DEFAULT is materialized by
+ // ParquetColumnVector as that default, not as null. The predicate must therefore be evaluated
+ // against the default. Evaluating against null could skip a whole file whose rows all match.
+ val defaultValue = 7L
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(defaultValue, LongType))
+ val expr = BloomFilterMightContain(
+ Literal(bloomBytes(bf), BinaryType),
+ new XxHash64(Seq(BoundReference(0, LongType, nullable = true))))
+ // Substituting the default keeps the file, because the default's hash is in the bloom.
+ assert(ParquetStorageFilter.create(Seq(expr), requested)
+ .rewriteForMissingKeys(Array(0), Array(defaultValue)).evalAllMissing(),
+ "substituting the existence default must probe the default's hash and keep the file")
+ // Substituting null instead would drop it, the bug this guards against. A second filter,
+ // because a rewrite is cached per set of missing positions: one scan always substitutes the
+ // same values for them.
+ assert(!ParquetStorageFilter.create(Seq(expr), requested)
+ .rewriteForMissingKeys(Array(0), Array(null)).evalAllMissing(),
+ "sanity check: substituting null probes a different hash and would drop the file")
+ }
+
+ test("rewriteForMissingKeys: null-safe equi-join bloom keeps the file") {
+ // Null-safe equi-join: ExtractEquiJoinKeys rewrites `a <=> b` so the join key becomes
+ // Coalesce(key, default). After rewriteForMissingKeys substitutes null for the
+ // BoundReference, the coalesce produces `default` (0L here) and the bloom probe checks
+ // whether the default's hash is in the bloom (which it is, matching what the creation side
+ // would have inserted for its own null rows). Expected result: keep the file.
+ val defaultHash = xxHash64(0L, LongType)
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(defaultHash)
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ val expr = BloomFilterMightContain(
+ bloomLit,
+ new XxHash64(Seq(Coalesce(Seq(
+ BoundReference(0, LongType, nullable = true),
+ Literal.default(LongType))))))
+
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(expr), requested)
+
+ val rewritten = filter.rewriteForMissingKeys(Array(0), Array(null))
+ assert(rewritten.keyColumnIndices.isEmpty,
+ "all key positions are missing; keyColumnIndices should be empty")
+ assert(rewritten.evalAllMissing(),
+ "Coalesce(null, default) yields default; bloom hit, so evalAllMissing must return true")
+ }
+
+ test("rewriteForMissingKeys: partial-missing keys narrow keyColumnIndices and renumber") {
+ // Two keys, one missing, one present. Verify the present key's BoundReference is renumbered to
+ // position 0 in the new layout, and the missing one is substituted with Literal(null). We don't
+ // run the predicate here, only the rewrite's structure.
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(1L, LongType))
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ // Two conjuncts: one on ordinal 0 (missing), one on ordinal 1 (present).
+ val exprs = Seq(
+ BloomFilterMightContain(
+ bloomLit, new XxHash64(Seq(BoundReference(0, LongType, nullable = true)))),
+ BloomFilterMightContain(
+ bloomLit, new XxHash64(Seq(BoundReference(1, LongType, nullable = true)))))
+
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(exprs, requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1))
+
+ val rewritten = filter.rewriteForMissingKeys(Array(0), Array(null))
+ assert(rewritten.keyColumnIndices.toSeq == Seq(1),
+ "only the present key column should remain in keyColumnIndices")
+ val ordinals = rewritten.boundExpression.collect {
+ case b: BoundReference => b.ordinal
+ }
+ assert(ordinals == Seq(0),
+ "the remaining BoundReference (for ordinal 1 in requested schema) should be " +
+ s"renumbered to local position 0; got $ordinals")
+ }
+
+ // Returns the FileSourceScanExec for a parquet read, with `storageFilters` attached. Goes through
+ // Spark's normal planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind, the SQL conf check,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def scanWithStorageFilter(
+ path: String,
+ keyName: String,
+ threshold: Long): FileSourceScanExec = {
+ val df = spark.read.parquet(path).select("k", "v")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == keyName).getOrElse(fail(s"No $keyName in scan output"))
+ val expr = GreaterThanOrEqual(keyAttr, Literal(threshold))
+ scan.copy(storageFilters = Seq(expr))
+ }
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect().map(r => (r.getLong(0), r.getString(1)))
+ }
+
+ test("end-to-end via FileSourceScanExec: conf on, filter applied, metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 195L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sortBy(_._1)}; expected $expected")
+
+ val rgSkipped = scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose: phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ assert(scan.simpleString(200).contains("GreaterThan(v,"),
+ s"the data filter must actually be pushed for this test to mean anything: " +
+ scan.simpleString(200))
+ // Storage filter on `k` (key column).
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(
+ storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(100L))))
+
+ val collected = executePlanCollect(withSF).toSet
+ // Every row satisfies v > 'v_000', so the storage filter alone decides the result.
+ val expected = rows.filter(_._1 >= 100L).toSet
+ assert(collected == expected,
+ s"got ${collected.toSeq.sortBy(_._1)}; expected ${expected.toSeq.sortBy(_._1)}")
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ plan.collect {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ plan.collect {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables (bf1: large, bf2: small with selective filter) for join tests.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 = "large" application side: 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 = "small" creation side: 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ body
+ }
+ }
+
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms >= 1,
+ s"expected >= 1 bloom filter on scan.storageFilters; got $storageBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune: the reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSet == Set(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves bloom filter as post-scan FilterExec when conf is off") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters when conf is off; got $storageBlooms")
+ assert(postScanBlooms >= 1,
+ s"expected bloom in post-scan FilterExec when conf is off; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("the feature still engages when ignoreCorruptFiles is on") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSet == Set((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since the
+ // post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty).getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+
+ test("FileSourceStrategy extraction preserves query results") {
+ withBloomFilterTables {
+ val baseConf = Map(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false")
+ val resultConfOff = withSQLConf(
+ (baseConf + (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false")).toSeq: _*) {
+ runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet
+ }
+ val resultConfOn = withSQLConf(
+ (baseConf + (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true")).toSeq: _*) {
+ runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet
+ }
+ assert(resultConfOn == resultConfOff,
+ s"results differ between conf-on and conf-off: on=$resultConfOn off=$resultConfOff")
+ }
+ }
+
+ Seq(Seq("k", "v"), Seq("k")).foreach { projection =>
+ test("a row group over the splice cap is read the plain way, same rows, more bytes " +
+ s"(projection ${projection.mkString(",")})") {
+ // Past the cap a row group is read the plain way: phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Two things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. And
+ // the fallback must actually have happened, which is observable: it reads the key column a
+ // second time, so it transfers strictly more bytes. Without that second assertion the test
+ // would pass even if the cap never reached the reader.
+ //
+ // The batch size is what makes the cap reachable at all: the reader examines the count only
+ // once a batch worth of survivors per key column has been buffered, so 51 survivors have to
+ // cross that line more than once.
+ //
+ // The `k` projection is the interesting one: an all-keys projection normally skips phase 2
+ // entirely, so past the cap it has to read the key column there like any other column.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val readSchema = StructType(projection.map(fileSchema(_)))
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, fileSchema, new StructType(), readSchema, Nil, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, Map.empty)
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = readerFn(file).asInstanceOf[Iterator[Object]].map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the
+ // row ranges: `k >= 350` keeps a contiguous run, so there is one range to hold.
+ val (plainRows, plainBytes) = run("100b")
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, fewer than the capacity, so the cap is never weighed and the row
+ // group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = ParquetStorageFilter.create(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true))),
+ StorageFilterMetrics(rowGroupsSkipped = rgSkipped),
+ maxSplicedRowGroupBytes = 100L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity = 16)
+ try {
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(rgSkipped.value >= 1,
+ s"expected a row group with no survivor at all; got ${rgSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group past the cap charges its second key read against the byte metric") {
+ // Giving splicing up means phase 2 reads the key columns a second time, while the baseline
+ // counts them once, in phase 1. That extra read is a cost against the saving, so the same file
+ // and filter must report less avoided than they do while splicing. Without that term the metric
+ // would credit the feature with bytes it did transfer.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+
+ def avoidedBytes(cap: Long): Long = {
+ val pf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = ParquetStorageFilter.create(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L))),
+ fileSchema, StorageFilterMetrics(bytesAvoidedByPageFiltering = pf), cap)
+ val (_, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => b.column(0).getLong(i), capacity = 16)
+ try pf.value finally reader.close()
+ }
+
+ val spliced = avoidedBytes(64L * 1024 * 1024)
+ val plain = avoidedBytes(100L)
+ assert(spliced > 0, s"page filtering has to avoid something here; got $spliced")
+ assert(plain < spliced,
+ s"the second key read must count against the saving; plain=$plain spliced=$spliced")
+ }
+ }
+
+ test("a row group whose survivors scatter into too many ranges is read without the filter") {
+ // `finalRanges` is built one row at a time, so an alternating filter makes one range per
+ // surviving row, and every column reader phase 2 drives materializes that list for the row
+ // group. Past the cap the filter is given up for that row group and every one of its rows is
+ // emitted, which the post-scan Filter then narrows.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def rowsWithCap(cap: Long, capacity: Int = 4096): Int = {
+ val filter =
+ ParquetStorageFilter.create(Seq(everyOtherRow), fileSchema, StorageFilterMetrics(), cap)
+ val (emitted, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.getRow(i).getLong(0), capacity)
+ try emitted.size finally reader.close()
+ }
+
+ assert(rowsWithCap(64L * 1024 * 1024) == 200,
+ "with room for the ranges, only the surviving rows come back")
+ // The budget is weighed in two places, and the capacity decides which one fires. At 4096 the
+ // 200 survivors never fill an accumulator, so only the check after phase 1 can catch them. At
+ // 16 the in-loop check is reached first, and it has to abandon the buffer and give the filter
+ // up together: keeping one without the other would leave phase 2 reading non-key columns
+ // while emit spliced keys from a queue holding only the first survivors.
+ assert(rowsWithCap(1024L) == 400,
+ "past the cap after phase 1, the filter is given up and the row group comes back whole")
+ assert(rowsWithCap(1024L, capacity = 16) == 400,
+ "and the same past the cap inside phase 1's survivor loop")
+ }
+ }
+
+ test("ANSI mode: a bloom over a cast join key stays in the post-scan Filter") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while a reader would evaluate
+ // it on every row of the ranges the pushed filter left. Extraction therefore has to decline, or
+ // this query would start failing when the conf is turned on.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it: a separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"a cast key must not be extracted.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSet == Set("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // `ParquetStorageFilter.test` evaluates the predicate without calling
+ // `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a
+ // `Nondeterministic` expression, so a non-deterministic conjunct has to stay behind. No
+ // producer builds one today, hence the hand-built plan: `InjectRuntimeFilter`'s blooms hash
+ // join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(42L, LongType))
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // Control: the same bloom over the key alone is offered to the scan, so the arms differ in
+ // exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. Two gates reject
+ // this one now: the planner's `deterministic` test, and `canEvaluateUnconditionally` inside
+ // the format's answer, whose whitelist is deterministic expressions only.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyOnlyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for: a dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`: `/` is floating-point division in Spark SQL, which would hand every key
+ // expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k"), rowGroupSize,
+ dictionary = dictionary)
+ }
+
+ // The encodings parquet actually used, over every column chunk of the file. Asserted rather than
+ // assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String): Set[Encoding] = {
+ val reader = ParquetFileReader.open(spark.sessionState.newHadoopConf(), new Path(filePath))
+ try {
+ reader.getFooter.getBlocks.asScala
+ .flatMap(_.getColumns.asScala)
+ .flatMap(_.getEncodings.asScala)
+ .toSet
+ } finally {
+ reader.close()
+ }
+ }
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // selects the plain (non-splicing) vectorized path. Every read helper in this suite goes through
+ // here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need: the caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } { reader =>
+ reader.setStorageFilter(storageFilter)
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // Renders one column value as a string using its *internal* representation, so the splicing path
+ // and the plain path can be compared without going through external type conversion.
+ private def renderValue(vec: ColumnVector, i: Int, dt: DataType): String = {
+ if (vec.isNullAt(i)) {
+ "null"
+ } else {
+ dt match {
+ case BooleanType => vec.getBoolean(i).toString
+ case ByteType => vec.getByte(i).toString
+ case ShortType => vec.getShort(i).toString
+ case IntegerType | DateType | _: YearMonthIntervalType => vec.getInt(i).toString
+ case LongType | TimestampType | TimestampNTZType | _: TimeType |
+ _: DayTimeIntervalType => vec.getLong(i).toString
+ case FloatType => vec.getFloat(i).toString
+ case DoubleType => vec.getDouble(i).toString
+ case d: DecimalType => vec.getDecimal(i, d.precision, d.scale).toString
+ case _: StringType => vec.getUTF8String(i).toString
+ case BinaryType => vec.getBinary(i).mkString(",")
+ case other => fail(s"renderValue does not handle $other")
+ }
+ }
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path: whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.column(0), i, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage: one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why: there is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ } {
+ val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded"
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyOnlyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = StructType(Seq(StructField("k", dt, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, dt))
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // Regression test: TimeType is an AtomicType that passes every planning-time gate (it is
+ // batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a TIME
+ // join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyOnlyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = StructType(Seq(StructField("k", dt, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // This is the contract that keeps FileSourceStrategy, ParquetStorageFilter.create and
+ // VectorizedParquetRecordReader.copierFor in lockstep. A type admitted here but missing from
+ // copierFor turns a planning-time rejection into a task failure.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyOnlyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("two key columns: both accumulators stay aligned with each other and with the data column") {
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300 => a in [100, 150]
+ val bound = And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)),
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1), "both key columns should be recognized")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, and the emit path depends on
+ // that: it pairs the k-th key slot of the batch with key-row position k, which is the identity
+ // only while the list is ascending. Nothing else in the suite supplies an unsorted one, because
+ // every other fixture happens to mention its keys in column order. Production does not promise
+ // that: the conjuncts arrive in `afterScanFilters` order, which follows the join. Without the
+ // sort this returns `a` and `b` swapped, with no exception to notice.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // Same predicate as the test above, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // Exercises the `i < isKeyTopLevel.length` branch of the emit loop: the partition slot sits
+ // past the end of isKeyTopLevel and must come from persistentBatchColumns, not the key queues.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite: OffHeapColumnVector.close() frees
+ // the native buffer, so a double close or a read after close is a crash rather than stale
+ // data. capacity = 16 over 100 survivors forces 7 emits, each closing the previous emit's
+ // dequeued key vectors.
+ withTempDir { dir =>
+ val path = writeKeyOnlyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(0L))
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (1L to 100L), s"expected all 100 keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test(s"$mode vectors: row-group skipping and page filtering over a mixed projection") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = keyAtLeastFilter(195L, StorageFilterMetrics(
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val reader = new VectorizedParquetRecordReader(useOffHeap, 4096)
+ reader.setStorageFilter(filter)
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ while (i < batch.numRows()) {
+ collected += ((batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+ i += 1
+ }
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 195L),
+ s"expected exact filtering; got ${collected.map(_._1)}")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and the
+ // partially kept one. Both counters must be non-negative, and together positive.
+ assert(bytesAvoidedRg.value >= 0 && bytesAvoidedPf.value >= 0,
+ s"avoided-byte metrics must never go negative; got rg=${bytesAvoidedRg.value} " +
+ s"pf=${bytesAvoidedPf.value}")
+ assert(bytesAvoidedRg.value + bytesAvoidedPf.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // The per-emit ColumnarBatch is replaced on every nextBatch(), so a consumer holding on to an
+ // earlier getCurrentValue() would read the wrong vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new VectorizedParquetRecordReader(false, 8)
+ try {
+ reader.setStorageFilter(filter)
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution: a column missing from the physical file -----
+
+ test("non-key column missing from a file: byte-avoided metrics tolerate a missing offset index") {
+ // ColumnIndexStore returns a null OffsetIndex for a column that is in the (clipped) requested
+ // schema but absent from the row group, which is exactly what schema evolution produces. The
+ // avoided-bytes walk runs over the full requested schema on every row group of every file, so
+ // without a null guard this is an NPE on the first row group of the older file.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // Older file: (k, v). Newer file: (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect()
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSet
+ val expected = ((150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))).toSet
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // The avoided-bytes counters are what walk the offset index of the missing column. Their
+ // being populated and non-negative is the evidence that the walk ran and coped.
+ val bytesRg = withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value >= 0 && bytesPf.value >= 0,
+ s"avoided-byte metrics must never go negative; got rg=${bytesRg.value} " +
+ s"pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = spark.read.parquet(path).select("k", "v").queryExecution.executedPlan
+ .collect { case s: FileSourceScanExec => s }.head
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Set[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Set[(Long, Option[Long])]) = {
+ val df = spark.sql(s"SELECT id, k FROM $table")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(buildFilter(keyAttr)))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = scan.output.indexWhere(_.name == "id")
+ val kPos = scan.output.indexWhere(_.name == "k")
+ val collected = rowPlan.executeCollect()
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSet
+ (withSF, collected)
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test: the default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Set[(Long, Option[Long])] = Set((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Set((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got ${rejectsNull.toSeq.sorted}")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got ${acceptsNull.toSeq.sorted}")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect().map(r => (r.getLong(0), r.getLong(2))).toSet
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1)).toSet
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got " +
+ s"${collected.toSeq.sorted.take(5)}")
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select("k", "s")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(195L))))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect()
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSet
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k")).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sorted}; expected $expected")
+ }
+ }
+ }
+
+ // ----- Planner gates and the lost-filter invariant -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false: it may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ assert(!subclass.supportsStorageFilter(bloom), "a subclass must not claim support")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ // A key the reader could not evaluate on rows the plan would have excluded. In ANSI mode the
+ // cast throws on a string that is not a number, and the reader evaluates the predicate on every
+ // row the pushed filter left, including those an earlier conjunct would have dropped.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+ assert(!format.supportsStorageFilter(onCast), "a cast key can throw on rows the plan excluded")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The whole lost-filter safety argument rests on this gate: if the reader cannot do late
+ // materialization, the planner must NOT move the bloom out of the post-scan Filter.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one: every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, "k", threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSet
+ assert(keys == (1L to 50L).toSet,
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support ignores a non-empty storageFilters") {
+ // The default `FileFormat.buildReaderWithStorageFilters` body may ignore what it is handed,
+ // because the post-scan Filter still holds it. Unreachable in production, since the planner
+ // asks `supportsStorageFilter` first.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(1L)))
+ val e = intercept[Exception] {
+ new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration())
+ }
+ // This stub implements no reader at all, so the error it fails with is the delegation's, which
+ // is what shows the filters were not rejected.
+ assert(e.getMessage.contains("buildReader is not supported"), e.getMessage)
+ }
+
+ Seq(false, true).foreach { aqe =>
+ test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") {
+ // AQE is on by default in production, and it is where the bloom subquery is planned by
+ // PlanAdaptiveSubqueries rather than PlanSubqueries, which is the path
+ // preparedStorageFilters' ScalarSubquery materialization depends on.
+ withBloomFilterTables {
+ val baseConf = Map(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString)
+ def run(pushdown: Boolean): Set[(Long, Long)] = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet
+ }
+ val off = run(false)
+ val on = run(true)
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ }
+ }
+ }
+
+ test("canonicalization keeps a storage-filter scan distinct from a plain one") {
+ // storageFilters is in doCanonicalize and in the case class equality, which is what stops
+ // exchange and subquery reuse from serving one scan's result to the other. Reuse compares
+ // canonicalized plans, so a scan that filters must not match a plain scan of the same file, and
+ // two scans carrying the same filter must still match.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, "k", threshold = 25L)
+ val plain = withSF.copy(storageFilters = Nil)
+ assert(withSF != plain, "case class equality must take storageFilters into account")
+ assert(!withSF.sameResult(plain),
+ s"a filtering scan must not be reusable as a plain one:\n${withSF.canonicalized}\n" +
+ s"${plain.canonicalized}")
+ // A second, independently planned scan of the same file with the same filter still
+ // matches, so reuse is not disabled wholesale. Its key attribute carries a different
+ // exprId, which is what canonicalization normalizes away.
+ val sameSF = scanWithStorageFilter(path, "k", threshold = 25L)
+ assert(withSF.sameResult(sameSF),
+ s"two scans with the same storage filter must stay reusable:\n" +
+ s"${withSF.canonicalized}\n${sameSF.canonicalized}")
+ val otherSF = scanWithStorageFilter(path, "k", threshold = 30L)
+ assert(!withSF.sameResult(otherSF), "a different threshold is a different result")
+ }
+ }
+ }
+
+ test("whole-stage codegen off: the row-at-a-time path still applies the extracted bloom") {
+ // The planner's extraction gate is `fileFormat.supportBatch`, which does not look at
+ // whole-stage codegen, while `FileSourceScanExec.supportsColumnar` does. So with codegen off
+ // the bloom is still extracted, `returningBatch` is false, and the reader serves the spliced
+ // batch one row at a time. That is the only combination where the planner's gate is weaker
+ // than the runtime's.
+ withBloomFilterTables {
+ val baseConf = Map(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "false")
+ def run(pushdown: Boolean): (Set[(Long, Long)], Int, Boolean) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, result) = runBloomFilterJoin()
+ val columnar = plan.collect { case s: FileSourceScanExec => s.supportsColumnar }
+ (result.map(r => (r.getLong(0), r.getLong(1))).toSet,
+ countBloomFiltersInStorageFilters(plan), columnar.forall(_ == false))
+ }
+ val (off, _, _) = run(false)
+ val (on, storageBlooms, noColumnarScan) = run(true)
+ assert(storageBlooms >= 1, s"the bloom must still be extracted with codegen off; " +
+ s"got $storageBlooms")
+ assert(noColumnarScan, "with codegen off no scan should output columnar batches")
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ }
+ }
+
+ test("off-heap column vectors through the planner: spliced values survive the free") {
+ // Off-heap is where the vector lifecycle actually bites: the previous batch's key vectors are
+ // freed at the next nextBatch(), so a stale reference reads released native memory rather than
+ // old bytes. This drives it through the planner, where the batch also crosses
+ // ColumnarToRowExec.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.COLUMN_VECTOR_OFFHEAP_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 150L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 150L).toSet
+ assert(collected == expected,
+ s"got ${collected.size} rows; expected ${expected.size}. " +
+ s"first few: ${collected.toSeq.sortBy(_._1).take(3)}")
+ }
+ }
+ }
+
+ // ----- Page-level pushedFilterRanges (a strict subset of the row group) -----
+
+ test("pushed data filter narrows to a page subset: phase 1 stays aligned with the row indexes") {
+ // Every other fixture writes one page per column per row group, so column-index filtering can
+ // only ever drop whole row groups and `pushedFilterRanges` is always the entire block. That
+ // makes the phase 1 alignment hold trivially, that the r-th row readBatch delivers pairs with
+ // rowIndexIter.nextLong(). With a small page size the data filter narrows
+ // to a page subset, so the two sequences only agree if the pairing is actually correct.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // `v` is correlated with `k`, so a range predicate on v prunes pages, not whole row groups.
+ val df = spark.read.parquet(path).select("k", "v").filter("v >= 'v_0300'")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ assert(scan.simpleString(200).contains("GreaterThanOrEqual(v,"),
+ s"the data filter must be pushed: ${scan.simpleString(200)}")
+ val keyAttr = scan.output.find(_.name == "k").get
+ // Storage filter keeps a band that starts inside the data filter's surviving range.
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(350L))))
+
+ val collected = executePlanCollect(withSF).toSet
+ val expected = rows.filter(r => r._2 >= "v_0300" && r._1 >= 350L).toSet
+ assert(collected == expected,
+ s"got ${collected.size} rows; expected ${expected.size}. " +
+ s"first few: ${collected.toSeq.sortBy(_._1).take(3)}")
+ }
+ }
+ }
+
+ test("page-subset ranges keep the avoided-byte metrics non-negative") {
+ // The strict-subset branch of compressedBytesForRowRanges (offset index + dictionary page) only
+ // runs when pushedFilterRanges is narrower than the block, which needs a multi-page row group.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v >= 'v_0100'")
+ val scan = df.queryExecution.executedPlan
+ .collect { case s: FileSourceScanExec => s }.head
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(390L))))
+ val collected = executePlanCollect(withSF).toSet
+ assert(collected == rows.filter(_._1 >= 390L).toSet, s"got ${collected.size} rows")
+
+ val bytesRg = withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value >= 0 && bytesPf.value >= 0,
Review Comment:
This assertion can never fail: `SQLMetric.add` ignores negative values and `SQLMetric.value` cannot be negative. The same applies to the assertions around lines 1403 and 1490. So a regression that makes `nonKeyBaselineBytes - phase2Bytes` negative would silently add 0 and still pass.
In addition, the `MissingOffsetIndexException` catch/retry and `fileHasNoOffsetIndex` paths are not exercised by any test. The test around line 1443 hits `chunk == null` for a missing column before any offset-index lookup. The PR description says such a file cannot be written with parquet-mr, but the `ParquetFileWriter.writeDataPage` overloads without a row count use `OffsetIndexBuilder.getNoOpBuilder()`, so a low-level writer (like the helpers in `ParquetVectorizedSuite`) can produce a file without an offset index.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayOutputStream, File}
+import java.net.URI
+import java.time.LocalTime
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapreduce.Job
+import org.apache.parquet.column.Encoding
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetInputFormat, ParquetOutputFormat}
+
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, XxHash64}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FileSourceScanLike, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests the late-materialization path of [[VectorizedParquetRecordReader]] driven by a
+ * [[ParquetStorageFilter]]. Writes small multi-row-group parquet files, wires a hand-built filter
+ * into the reader, and asserts correctness + the two storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false), StructField("v", StringType, nullable = false)))
+ ParquetStorageFilter.create(Seq(expr), requested, metrics)
+ }
+
+ // Writes a single-column (just `k`) parquet file for the supplied key type via Spark's
+ // {@code Encoder}.
+ private def writeKeyOnlyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(dir, spark.createDataset(keys).toDF("k"), rowGroupSize)
+
+ // Reads a key-only file, returning the survivor keys and the reader. The {@code extract} function
+ // pulls one value at a time from the batch's key column.
+ private def readKeyOnlyAll[T](
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T,
+ capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k"), storageFilter, (batch, i) => extract(batch.column(0), i),
+ capacity)
+
+ // Builds a `k >= threshold` storage filter bound to position 0 against a key-only schema of the
+ // given key type.
+ private def keyOnlyAtLeastFilter(
+ threshold: Literal,
+ keyType: DataType,
+ metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, keyType, nullable = false), threshold)
+ val requested = StructType(Seq(StructField("k", keyType, nullable = false)))
+ ParquetStorageFilter.create(Seq(expr), requested, metrics)
+ }
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows in one row group: parquet's first row-group size check is at record 100, so
+ // `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = keyAtLeastFilter(1000L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf,
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(rgSkipped.value > 0,
+ s"expected at least one row group skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedRg.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${rowsExcludedRg.value}")
+ assert(rowsExcludedPf.value == 0,
+ s"no partial-row-group filtering expected; got ${rowsExcludedPf.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(bytesAvoidedRg.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got ${bytesAvoidedRg.value}")
+ assert(bytesAvoidedPf.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got ${bytesAvoidedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val filter = keyAtLeastFilter(0L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped, rowsExcludedWithinRowGroup = rowsExcludedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(rgSkipped.value == 0, s"nothing should be skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedPf.value == 0,
+ s"nothing should be filtered; got ${rowsExcludedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("mixed: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ // Output is exact: VectorizedColumnReader uses PageReadStore.getRowIndexes (driven by
+ // our finalRanges) to skip rows within partial pages, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it: a row group that was neither skipped whole nor emitted has
+ // to have its rows counted there.
+ assert(rowsExcludedPf.value > 0,
+ s"expected rows excluded inside a kept row group; got ${rowsExcludedPf.value}")
+ assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == rows.size,
+ s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " +
+ s"should account for all ${rows.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key-only projection: phase 2 is skipped and both byte-avoided metrics are zero") {
+ // When the projected schema contains only the bloom key, phase 2 is skipped entirely
+ // (`nonKeyColumns == null` in the reader). All output rows come from the per-key-column
+ // queues populated in phase 1. Total bytes read match the no-storage-filter path (phase 1 reads
+ // the key column once instead of phase 2 re-reading it), so both `avoided` metrics are zero:
+ // there are no non-key bytes to skip. This is the shape the design notes call the biggest win,
+ // so it must not be the shape that pays for metrics.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ // k >= 195 -> last 6 keys survive; preceding row groups skipped or page-pruned.
+ val filter = keyOnlyAtLeastFilter(Literal(195L), LongType, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf,
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i))
+ try {
+ val expected = keys.filter(_ >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact survivor keys; got ${result.sorted}, expected ${expected.toSeq.sorted}")
+ assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedRg.value > 0,
+ s"a skipped row group must count its rows too; got ${rowsExcludedRg.value}")
+ // The all-keys path takes its kept-row count from `finalRowCount` rather than from a
+ // phase-2 page store, so this identity is the only thing that checks that arithmetic.
+ assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == keys.size,
+ s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " +
+ s"should account for all ${keys.size} rows")
+ // For an all-keys projection, the no-storage-filter path would have read the same key
+ // column phase 1 reads. There are no non-key bytes to avoid; both metrics are 0.
+ assert(bytesAvoidedRg.value == 0,
+ s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedRg.value}")
+ assert(bytesAvoidedPf.value == 0,
+ s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("multi-batch emit: survivor count exceeds capacity") {
+ // Drive the reader at capacity = 16 with a row group of 100 surviving rows. Exercises:
+ // - The per-key-column queue holding multiple full-capacity vectors plus a partial tail.
+ // - pendingCloseKeyVectors getting closed at the start of every subsequent emit.
+ // - The per-emit ColumnarBatch reconstruction running ceil(100/16) = 7 times.
+ withTempDir { dir =>
+ val keys = (1L to 100L)
+ // Big rowGroupSize so all 100 rows fit in one row group.
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 64 * 1024L)
+ // Filter accepts every row so the queue is fully populated.
+ val filter = keyOnlyAtLeastFilter(Literal(0L), LongType)
+ val (result, reader) =
+ readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i), capacity = 16)
+ try {
+ assert(result == keys.toSeq,
+ s"expected all keys returned in order across multiple batches; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("int key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the IntegerType branch of ValueCopier.
+ withTempDir { dir =>
+ val keys = (1 to 100)
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyOnlyAtLeastFilter(Literal(90), IntegerType)
+ val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getInt(i))
+ try {
+ assert(result.toSet == keys.filter(_ >= 90).toSet,
+ s"expected int keys >= 90; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("string key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the StringType branch of ValueCopier (variable-length byte copy via putByteArray).
+ withTempDir { dir =>
+ val keys = (1 to 20).map(i => f"k$i%03d")
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyOnlyAtLeastFilter(
+ Literal.create("k015", StringType), StringType)
+ val (result, reader) =
+ readKeyOnlyAll(path, filter, (vec, i) => vec.getUTF8String(i).toString)
+ try {
+ assert(result.toSet == keys.filter(_ >= "k015").toSet,
+ s"expected string keys >= 'k015'; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction: extractStorageFilters pre-checks each one, and
+ // by the time create runs the conjunct is gone from the post-scan Filter, so a soft rejection
+ // would silently return rows the filter excludes. create fails instead.
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+
+ // Nothing to push: the caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+ }
+
+ // Serializes a [[BloomFilter]] to bytes suitable for a [[Literal]].
+ private def bloomBytes(bf: BloomFilter): Array[Byte] = {
+ val out = new ByteArrayOutputStream()
+ bf.writeTo(out)
+ out.toByteArray
+ }
+
+ // Computes `XxHash64(v)` using the same seed Spark uses for runtime bloom filters.
+ private def xxHash64(v: Any, dt: DataType): Long = {
+ new XxHash64(Seq(Literal(v, dt))).eval(InternalRow.empty).asInstanceOf[Long]
+ }
+
+ test("rewriteForMissingKeys: regular equi-join bloom probes the null key's hash") {
+ // Regular equi-join: the runtime bloom is BloomFilterMightContain(bloom, XxHash64(key)).
+ // XxHash64 is a HashExpression, so it is nullable = false and hashes a null input to its SEED
+ // rather than producing null. Substituting Literal(null) therefore leaves a concrete probe for
+ // `xxHash64(null)`, and whether the file is kept depends on whether that hash is in the bloom.
+ // Both directions are asserted so the outcome does not hinge on a lucky bloom miss.
+ val nullKeyHash = xxHash64(null, LongType)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+
+ def rewriteWithBloomContaining(hashes: Long*): ParquetStorageFilter = {
+ val bf = BloomFilter.create(10, 128)
+ hashes.foreach(bf.putLong)
+ val expr = BloomFilterMightContain(
+ Literal(bloomBytes(bf), BinaryType),
+ new XxHash64(Seq(BoundReference(0, LongType, nullable = true))))
+ val filter = ParquetStorageFilter.create(Seq(expr), requested)
+ filter.rewriteForMissingKeys(Array(0), Array(null))
+ }
+
+ val dropped = rewriteWithBloomContaining(xxHash64(42L, LongType))
+ assert(dropped.keyColumnIndices.isEmpty,
+ "all key positions are missing; keyColumnIndices should be empty")
+ assert(!dropped.evalAllMissing(),
+ "the null key's hash is not in the bloom, so evalAllMissing must return false " +
+ "(skip the file)")
+
+ val kept = rewriteWithBloomContaining(nullKeyHash)
+ assert(kept.evalAllMissing(),
+ "the null key's hash IS in the bloom, so evalAllMissing must return true (keep the file)")
+ }
+
+ test("rewriteForMissingKeys: missing key with an existence DEFAULT probes the default's hash") {
+ // A missing column that has a non-null existence DEFAULT is materialized by
+ // ParquetColumnVector as that default, not as null. The predicate must therefore be evaluated
+ // against the default. Evaluating against null could skip a whole file whose rows all match.
+ val defaultValue = 7L
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(defaultValue, LongType))
+ val expr = BloomFilterMightContain(
+ Literal(bloomBytes(bf), BinaryType),
+ new XxHash64(Seq(BoundReference(0, LongType, nullable = true))))
+ // Substituting the default keeps the file, because the default's hash is in the bloom.
+ assert(ParquetStorageFilter.create(Seq(expr), requested)
+ .rewriteForMissingKeys(Array(0), Array(defaultValue)).evalAllMissing(),
+ "substituting the existence default must probe the default's hash and keep the file")
+ // Substituting null instead would drop it, the bug this guards against. A second filter,
+ // because a rewrite is cached per set of missing positions: one scan always substitutes the
+ // same values for them.
+ assert(!ParquetStorageFilter.create(Seq(expr), requested)
+ .rewriteForMissingKeys(Array(0), Array(null)).evalAllMissing(),
+ "sanity check: substituting null probes a different hash and would drop the file")
+ }
+
+ test("rewriteForMissingKeys: null-safe equi-join bloom keeps the file") {
+ // Null-safe equi-join: ExtractEquiJoinKeys rewrites `a <=> b` so the join key becomes
+ // Coalesce(key, default). After rewriteForMissingKeys substitutes null for the
+ // BoundReference, the coalesce produces `default` (0L here) and the bloom probe checks
+ // whether the default's hash is in the bloom (which it is, matching what the creation side
+ // would have inserted for its own null rows). Expected result: keep the file.
+ val defaultHash = xxHash64(0L, LongType)
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(defaultHash)
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ val expr = BloomFilterMightContain(
+ bloomLit,
+ new XxHash64(Seq(Coalesce(Seq(
+ BoundReference(0, LongType, nullable = true),
+ Literal.default(LongType))))))
+
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(expr), requested)
+
+ val rewritten = filter.rewriteForMissingKeys(Array(0), Array(null))
+ assert(rewritten.keyColumnIndices.isEmpty,
+ "all key positions are missing; keyColumnIndices should be empty")
+ assert(rewritten.evalAllMissing(),
+ "Coalesce(null, default) yields default; bloom hit, so evalAllMissing must return true")
+ }
+
+ test("rewriteForMissingKeys: partial-missing keys narrow keyColumnIndices and renumber") {
+ // Two keys, one missing, one present. Verify the present key's BoundReference is renumbered to
+ // position 0 in the new layout, and the missing one is substituted with Literal(null). We don't
+ // run the predicate here, only the rewrite's structure.
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(1L, LongType))
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ // Two conjuncts: one on ordinal 0 (missing), one on ordinal 1 (present).
+ val exprs = Seq(
+ BloomFilterMightContain(
+ bloomLit, new XxHash64(Seq(BoundReference(0, LongType, nullable = true)))),
+ BloomFilterMightContain(
+ bloomLit, new XxHash64(Seq(BoundReference(1, LongType, nullable = true)))))
+
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(exprs, requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1))
+
+ val rewritten = filter.rewriteForMissingKeys(Array(0), Array(null))
+ assert(rewritten.keyColumnIndices.toSeq == Seq(1),
+ "only the present key column should remain in keyColumnIndices")
+ val ordinals = rewritten.boundExpression.collect {
+ case b: BoundReference => b.ordinal
+ }
+ assert(ordinals == Seq(0),
+ "the remaining BoundReference (for ordinal 1 in requested schema) should be " +
+ s"renumbered to local position 0; got $ordinals")
+ }
+
+ // Returns the FileSourceScanExec for a parquet read, with `storageFilters` attached. Goes through
+ // Spark's normal planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind, the SQL conf check,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def scanWithStorageFilter(
+ path: String,
+ keyName: String,
+ threshold: Long): FileSourceScanExec = {
+ val df = spark.read.parquet(path).select("k", "v")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == keyName).getOrElse(fail(s"No $keyName in scan output"))
+ val expr = GreaterThanOrEqual(keyAttr, Literal(threshold))
+ scan.copy(storageFilters = Seq(expr))
+ }
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect().map(r => (r.getLong(0), r.getString(1)))
+ }
+
+ test("end-to-end via FileSourceScanExec: conf on, filter applied, metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 195L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sortBy(_._1)}; expected $expected")
+
+ val rgSkipped = scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose: phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ assert(scan.simpleString(200).contains("GreaterThan(v,"),
+ s"the data filter must actually be pushed for this test to mean anything: " +
+ scan.simpleString(200))
+ // Storage filter on `k` (key column).
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(
+ storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(100L))))
+
+ val collected = executePlanCollect(withSF).toSet
+ // Every row satisfies v > 'v_000', so the storage filter alone decides the result.
+ val expected = rows.filter(_._1 >= 100L).toSet
+ assert(collected == expected,
+ s"got ${collected.toSeq.sortBy(_._1)}; expected ${expected.toSeq.sortBy(_._1)}")
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ plan.collect {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ plan.collect {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables (bf1: large, bf2: small with selective filter) for join tests.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 = "large" application side: 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 = "small" creation side: 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ body
+ }
+ }
+
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms >= 1,
+ s"expected >= 1 bloom filter on scan.storageFilters; got $storageBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune: the reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSet == Set(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves bloom filter as post-scan FilterExec when conf is off") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters when conf is off; got $storageBlooms")
+ assert(postScanBlooms >= 1,
+ s"expected bloom in post-scan FilterExec when conf is off; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("the feature still engages when ignoreCorruptFiles is on") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSet == Set((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since the
+ // post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty).getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+
+ test("FileSourceStrategy extraction preserves query results") {
+ withBloomFilterTables {
+ val baseConf = Map(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false")
+ val resultConfOff = withSQLConf(
+ (baseConf + (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false")).toSeq: _*) {
+ runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet
+ }
+ val resultConfOn = withSQLConf(
+ (baseConf + (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true")).toSeq: _*) {
+ runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet
+ }
+ assert(resultConfOn == resultConfOff,
+ s"results differ between conf-on and conf-off: on=$resultConfOn off=$resultConfOff")
+ }
+ }
+
+ Seq(Seq("k", "v"), Seq("k")).foreach { projection =>
+ test("a row group over the splice cap is read the plain way, same rows, more bytes " +
+ s"(projection ${projection.mkString(",")})") {
+ // Past the cap a row group is read the plain way: phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Two things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. And
+ // the fallback must actually have happened, which is observable: it reads the key column a
+ // second time, so it transfers strictly more bytes. Without that second assertion the test
+ // would pass even if the cap never reached the reader.
+ //
+ // The batch size is what makes the cap reachable at all: the reader examines the count only
+ // once a batch worth of survivors per key column has been buffered, so 51 survivors have to
+ // cross that line more than once.
+ //
+ // The `k` projection is the interesting one: an all-keys projection normally skips phase 2
+ // entirely, so past the cap it has to read the key column there like any other column.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val readSchema = StructType(projection.map(fileSchema(_)))
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, fileSchema, new StructType(), readSchema, Nil, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, Map.empty)
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = readerFn(file).asInstanceOf[Iterator[Object]].map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the
+ // row ranges: `k >= 350` keeps a contiguous run, so there is one range to hold.
+ val (plainRows, plainBytes) = run("100b")
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, fewer than the capacity, so the cap is never weighed and the row
+ // group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = ParquetStorageFilter.create(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true))),
+ StorageFilterMetrics(rowGroupsSkipped = rgSkipped),
+ maxSplicedRowGroupBytes = 100L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity = 16)
+ try {
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(rgSkipped.value >= 1,
+ s"expected a row group with no survivor at all; got ${rgSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group past the cap charges its second key read against the byte metric") {
+ // Giving splicing up means phase 2 reads the key columns a second time, while the baseline
+ // counts them once, in phase 1. That extra read is a cost against the saving, so the same file
+ // and filter must report less avoided than they do while splicing. Without that term the metric
+ // would credit the feature with bytes it did transfer.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+
+ def avoidedBytes(cap: Long): Long = {
+ val pf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = ParquetStorageFilter.create(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L))),
+ fileSchema, StorageFilterMetrics(bytesAvoidedByPageFiltering = pf), cap)
+ val (_, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => b.column(0).getLong(i), capacity = 16)
+ try pf.value finally reader.close()
+ }
+
+ val spliced = avoidedBytes(64L * 1024 * 1024)
+ val plain = avoidedBytes(100L)
+ assert(spliced > 0, s"page filtering has to avoid something here; got $spliced")
+ assert(plain < spliced,
+ s"the second key read must count against the saving; plain=$plain spliced=$spliced")
+ }
+ }
+
+ test("a row group whose survivors scatter into too many ranges is read without the filter") {
+ // `finalRanges` is built one row at a time, so an alternating filter makes one range per
+ // surviving row, and every column reader phase 2 drives materializes that list for the row
+ // group. Past the cap the filter is given up for that row group and every one of its rows is
+ // emitted, which the post-scan Filter then narrows.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def rowsWithCap(cap: Long, capacity: Int = 4096): Int = {
+ val filter =
+ ParquetStorageFilter.create(Seq(everyOtherRow), fileSchema, StorageFilterMetrics(), cap)
+ val (emitted, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.getRow(i).getLong(0), capacity)
+ try emitted.size finally reader.close()
+ }
+
+ assert(rowsWithCap(64L * 1024 * 1024) == 200,
+ "with room for the ranges, only the surviving rows come back")
+ // The budget is weighed in two places, and the capacity decides which one fires. At 4096 the
+ // 200 survivors never fill an accumulator, so only the check after phase 1 can catch them. At
+ // 16 the in-loop check is reached first, and it has to abandon the buffer and give the filter
+ // up together: keeping one without the other would leave phase 2 reading non-key columns
+ // while emit spliced keys from a queue holding only the first survivors.
+ assert(rowsWithCap(1024L) == 400,
+ "past the cap after phase 1, the filter is given up and the row group comes back whole")
+ assert(rowsWithCap(1024L, capacity = 16) == 400,
+ "and the same past the cap inside phase 1's survivor loop")
+ }
+ }
+
+ test("ANSI mode: a bloom over a cast join key stays in the post-scan Filter") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while a reader would evaluate
+ // it on every row of the ranges the pushed filter left. Extraction therefore has to decline, or
+ // this query would start failing when the conf is turned on.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it: a separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"a cast key must not be extracted.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSet == Set("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // `ParquetStorageFilter.test` evaluates the predicate without calling
+ // `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a
+ // `Nondeterministic` expression, so a non-deterministic conjunct has to stay behind. No
+ // producer builds one today, hence the hand-built plan: `InjectRuntimeFilter`'s blooms hash
+ // join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(42L, LongType))
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // Control: the same bloom over the key alone is offered to the scan, so the arms differ in
+ // exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. Two gates reject
+ // this one now: the planner's `deterministic` test, and `canEvaluateUnconditionally` inside
+ // the format's answer, whose whitelist is deterministic expressions only.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyOnlyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for: a dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`: `/` is floating-point division in Spark SQL, which would hand every key
+ // expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k"), rowGroupSize,
+ dictionary = dictionary)
+ }
+
+ // The encodings parquet actually used, over every column chunk of the file. Asserted rather than
+ // assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String): Set[Encoding] = {
+ val reader = ParquetFileReader.open(spark.sessionState.newHadoopConf(), new Path(filePath))
+ try {
+ reader.getFooter.getBlocks.asScala
+ .flatMap(_.getColumns.asScala)
+ .flatMap(_.getEncodings.asScala)
+ .toSet
+ } finally {
+ reader.close()
+ }
+ }
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // selects the plain (non-splicing) vectorized path. Every read helper in this suite goes through
+ // here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need: the caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } { reader =>
+ reader.setStorageFilter(storageFilter)
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // Renders one column value as a string using its *internal* representation, so the splicing path
+ // and the plain path can be compared without going through external type conversion.
+ private def renderValue(vec: ColumnVector, i: Int, dt: DataType): String = {
+ if (vec.isNullAt(i)) {
+ "null"
+ } else {
+ dt match {
+ case BooleanType => vec.getBoolean(i).toString
+ case ByteType => vec.getByte(i).toString
+ case ShortType => vec.getShort(i).toString
+ case IntegerType | DateType | _: YearMonthIntervalType => vec.getInt(i).toString
+ case LongType | TimestampType | TimestampNTZType | _: TimeType |
+ _: DayTimeIntervalType => vec.getLong(i).toString
+ case FloatType => vec.getFloat(i).toString
+ case DoubleType => vec.getDouble(i).toString
+ case d: DecimalType => vec.getDecimal(i, d.precision, d.scale).toString
+ case _: StringType => vec.getUTF8String(i).toString
+ case BinaryType => vec.getBinary(i).mkString(",")
+ case other => fail(s"renderValue does not handle $other")
+ }
+ }
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path: whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.column(0), i, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage: one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why: there is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ } {
+ val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded"
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyOnlyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = StructType(Seq(StructField("k", dt, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, dt))
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // Regression test: TimeType is an AtomicType that passes every planning-time gate (it is
+ // batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a TIME
+ // join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyOnlyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = StructType(Seq(StructField("k", dt, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // This is the contract that keeps FileSourceStrategy, ParquetStorageFilter.create and
+ // VectorizedParquetRecordReader.copierFor in lockstep. A type admitted here but missing from
+ // copierFor turns a planning-time rejection into a task failure.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyOnlyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("two key columns: both accumulators stay aligned with each other and with the data column") {
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300 => a in [100, 150]
+ val bound = And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)),
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1), "both key columns should be recognized")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, and the emit path depends on
+ // that: it pairs the k-th key slot of the batch with key-row position k, which is the identity
+ // only while the list is ascending. Nothing else in the suite supplies an unsorted one, because
+ // every other fixture happens to mention its keys in column order. Production does not promise
+ // that: the conjuncts arrive in `afterScanFilters` order, which follows the join. Without the
+ // sort this returns `a` and `b` swapped, with no exception to notice.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // Same predicate as the test above, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // Exercises the `i < isKeyTopLevel.length` branch of the emit loop: the partition slot sits
+ // past the end of isKeyTopLevel and must come from persistentBatchColumns, not the key queues.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite: OffHeapColumnVector.close() frees
+ // the native buffer, so a double close or a read after close is a crash rather than stale
+ // data. capacity = 16 over 100 survivors forces 7 emits, each closing the previous emit's
+ // dequeued key vectors.
+ withTempDir { dir =>
+ val path = writeKeyOnlyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(0L))
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (1L to 100L), s"expected all 100 keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test(s"$mode vectors: row-group skipping and page filtering over a mixed projection") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = keyAtLeastFilter(195L, StorageFilterMetrics(
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val reader = new VectorizedParquetRecordReader(useOffHeap, 4096)
+ reader.setStorageFilter(filter)
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ while (i < batch.numRows()) {
+ collected += ((batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+ i += 1
+ }
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 195L),
+ s"expected exact filtering; got ${collected.map(_._1)}")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and the
+ // partially kept one. Both counters must be non-negative, and together positive.
+ assert(bytesAvoidedRg.value >= 0 && bytesAvoidedPf.value >= 0,
+ s"avoided-byte metrics must never go negative; got rg=${bytesAvoidedRg.value} " +
+ s"pf=${bytesAvoidedPf.value}")
+ assert(bytesAvoidedRg.value + bytesAvoidedPf.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // The per-emit ColumnarBatch is replaced on every nextBatch(), so a consumer holding on to an
+ // earlier getCurrentValue() would read the wrong vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new VectorizedParquetRecordReader(false, 8)
+ try {
+ reader.setStorageFilter(filter)
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution: a column missing from the physical file -----
+
+ test("non-key column missing from a file: byte-avoided metrics tolerate a missing offset index") {
+ // ColumnIndexStore returns a null OffsetIndex for a column that is in the (clipped) requested
+ // schema but absent from the row group, which is exactly what schema evolution produces. The
+ // avoided-bytes walk runs over the full requested schema on every row group of every file, so
+ // without a null guard this is an NPE on the first row group of the older file.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // Older file: (k, v). Newer file: (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect()
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSet
+ val expected = ((150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))).toSet
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // The avoided-bytes counters are what walk the offset index of the missing column. Their
+ // being populated and non-negative is the evidence that the walk ran and coped.
+ val bytesRg = withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value >= 0 && bytesPf.value >= 0,
+ s"avoided-byte metrics must never go negative; got rg=${bytesRg.value} " +
+ s"pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = spark.read.parquet(path).select("k", "v").queryExecution.executedPlan
+ .collect { case s: FileSourceScanExec => s }.head
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Set[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Set[(Long, Option[Long])]) = {
+ val df = spark.sql(s"SELECT id, k FROM $table")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(buildFilter(keyAttr)))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = scan.output.indexWhere(_.name == "id")
+ val kPos = scan.output.indexWhere(_.name == "k")
+ val collected = rowPlan.executeCollect()
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSet
+ (withSF, collected)
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test: the default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Set[(Long, Option[Long])] = Set((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Set((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got ${rejectsNull.toSeq.sorted}")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got ${acceptsNull.toSeq.sorted}")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect().map(r => (r.getLong(0), r.getLong(2))).toSet
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1)).toSet
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got " +
+ s"${collected.toSeq.sorted.take(5)}")
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select("k", "s")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(195L))))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect()
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSet
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k")).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sorted}; expected $expected")
+ }
+ }
+ }
+
+ // ----- Planner gates and the lost-filter invariant -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false: it may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ assert(!subclass.supportsStorageFilter(bloom), "a subclass must not claim support")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ // A key the reader could not evaluate on rows the plan would have excluded. In ANSI mode the
+ // cast throws on a string that is not a number, and the reader evaluates the predicate on every
+ // row the pushed filter left, including those an earlier conjunct would have dropped.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+ assert(!format.supportsStorageFilter(onCast), "a cast key can throw on rows the plan excluded")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The whole lost-filter safety argument rests on this gate: if the reader cannot do late
+ // materialization, the planner must NOT move the bloom out of the post-scan Filter.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one: every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, "k", threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSet
+ assert(keys == (1L to 50L).toSet,
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support ignores a non-empty storageFilters") {
+ // The default `FileFormat.buildReaderWithStorageFilters` body may ignore what it is handed,
+ // because the post-scan Filter still holds it. Unreachable in production, since the planner
+ // asks `supportsStorageFilter` first.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(1L)))
+ val e = intercept[Exception] {
+ new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration())
+ }
+ // This stub implements no reader at all, so the error it fails with is the delegation's, which
+ // is what shows the filters were not rejected.
+ assert(e.getMessage.contains("buildReader is not supported"), e.getMessage)
+ }
+
+ Seq(false, true).foreach { aqe =>
Review Comment:
This is the only join-based test running with AQE on, and it only checks `on == off` and `on.nonEmpty`. Since the post-scan `Filter` keeps the bloom conjunct, the results are identical whether or not the storage filter is actually applied. All the tests that check the storage-filter count or metrics set `ADAPTIVE_EXECUTION_ENABLED` to `false`, and `countBloomFiltersInStorageFilters` uses `plan.collect`, which doesn't descend into `AdaptiveSparkPlanExec`. So a regression where AQE (e.g., `PlanAdaptiveSubqueries` or re-optimization) silently drops the storage filter would pass all tests. Could we add an AQE-on assertion on the final plan (e.g., via `collectWithSubqueries`/`stripAQEPlan`) or on the metrics?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -771,6 +833,28 @@ case class FileSourceScanExec(
readRDD
}
+ // Materialize scalar subqueries inside storage filters to literals and bind AttributeReferences
+ // to BoundReferences targeting positions in `requiredSchema`. Subqueries must have been prepared
+ // by SparkPlan before this is forced (same contract as `pushedDownFilters`).
+ @transient
+ protected lazy val preparedStorageFilters: Seq[Expression] = {
+ if (storageFilters.isEmpty) {
+ Nil
+ } else {
+ // No conf check here: the conf decides at planning time whether a scan is offered storage
+ // filters at all, and re-reading it now could only make this scan drop work it already has.
+ //
+ // `output` is `readDataColumns ++ generatedMetadataColumns ++ partitionColumns ++
+ // constantMetadataColumns` and `requiredSchema` is the StructType of the first two groups, so
+ // the first `requiredSchema.length` attributes line up with its fields.
+ val requestedDataAttrs = output.take(requiredSchema.length)
+ storageFilters.map { expr =>
+ val subqueryReplaced = expr.transform { case s: execution.ScalarSubquery => s.toLiteral }
Review Comment:
This embeds the materialized bloom filter (up to 8MiB with the default `spark.sql.optimizer.runtime.bloomFilter.maxNumBits`) as a binary `Literal` into `ParquetStorageFilter`, which is captured by the reader closure of `FileScanRDD`. Since it is a distinct object from the copy the post-scan `Filter` already ships, the stage task binary grows by the bloom size, and each task deserializes the literal and then deserializes it again into a `BloomFilter` via `Predicate.create` (and once more per `rewriteForMissingKeys`). That is roughly 16MiB of extra heap per concurrent task. Could we broadcast the bloom bytes once (like `InSubqueryExec.resultBroadcast`) and share the deserialized filter per executor?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -151,6 +151,59 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
}
}
+ /**
+ * The conjuncts of `afterScanFilters` the file format can evaluate at the storage layer for late
+ * materialization, to prune value-column IO.
+ *
+ * They stay in the post-scan `Filter` as well, the way a pushed data filter does: the reader is
+ * offered them, not obliged to honor them, so the plan keeps the exact check. What it costs is
+ * evaluating the conjunct a second time for the rows the scan emits. What it buys is a reader
+ * free to give up on a file with no page index, or on a row group whose survivors scatter too far
+ * to hold their row ranges, without the answer depending on it.
+ *
+ * Two of the conditions are per scan, and failing either offers nothing:
+ * - The storage-filter pushdown SQL conf is on.
+ * - [[FileFormat.supportBatch]] holds for the schema the reader will see,
+ * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives
+ * its own vectorized-read decision from. Late materialization needs a batch read, so this asks
+ * about batch support rather than naming a format.
+ *
+ * The rest are per conjunct:
+ * - It is deterministic. A reader evaluates the predicate without
+ * `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a
+ * `Nondeterministic` expression, so a non-deterministic conjunct would fail at task time.
+ * - It references at least one column, and every column it references is a projected data
+ * column. A reference to something the scan does not read cannot be evaluated by the reader.
+ * - [[FileFormat.supportsStorageFilter]] accepts it. That is where the expression shapes and
+ * column types a reader can evaluate live, so this method names neither a format nor a type.
+ *
+ * One last condition is on the set that survives: at least one projected data column must be left
+ * for the reader to prune. A scan that projects nothing but the filter's own key columns reads
+ * the same columns for the same rows either way, since the reader has to read a key column to
+ * evaluate the filter on it, so offering it could only add the cost of evaluating the predicate
+ * outside the generated code. Nothing is offered in that case.
+ */
+ private def storageFiltersFor(
+ afterScanFilters: ExpressionSet,
+ fsRelation: HadoopFsRelation,
+ readDataColumns: Seq[Attribute],
+ outputDataSchema: StructType): Seq[Expression] = {
+ val sparkSession = fsRelation.sparkSession
+ val sqlConf = sparkSession.sessionState.conf
+ if (!sqlConf.parquetStorageFilterPushdownEnabled) return Nil
Review Comment:
The doc of this method says it names neither a format nor a type, but it reads the Parquet-named conf `spark.sql.parquet.storageFilterPushdown.enabled` in the format-agnostic strategy. Similarly, the generic `FileSourceScanLike` defines five Parquet-specific metrics (row groups, page filtering), and `ParquetFileFormat` imports the physical node's companion object to get their keys. Once a second format implements `supportsStorageFilter` (or the planned DSv2 step lands), it would be switched by a Parquet conf. Could we move the conf check into `ParquetFileFormat.supportsStorageFilter` (as `supportBatch` already reads Parquet confs inside the format) and let the format declare its metrics?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
Review Comment:
The absence of an offset index is known from the in-memory footer without any IO: `ColumnChunkMetaData.getOffsetIndexReference() == null`, which is exactly when `ParquetFileReader.readOffsetIndex` returns null and `ColumnIndexStoreImpl` throws `MissingOffsetIndexException`. Detecting it only here means:
- For every split of a file written without a page index (e.g., pyarrow's default `write_page_index=False`), the first partially-surviving row group pays the phase-1 buffering, a WARN log with a stack trace, and a full re-read of the projection including the keys.
- After that, `filterGivenUp = fileHasNoOffsetIndex` skips phase 1 for the rest of the split, so even whole-row-group skips, which don't need an offset index, are lost.
- With column-index filtering on and a pushed filter, a missing offset index on only a key column makes the init-time `ColumnIndexStore` for the block `EMPTY`, so the phase-2 read of non-key columns also throws and the filter is given up for the whole file, even though the non-key columns have offset indexes.
Also, the SQLConf doc sentence "since a missing page index is only reported by attempting the read" is not accurate. Could we check the footer up front instead?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -151,6 +151,59 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
}
}
+ /**
+ * The conjuncts of `afterScanFilters` the file format can evaluate at the storage layer for late
+ * materialization, to prune value-column IO.
+ *
+ * They stay in the post-scan `Filter` as well, the way a pushed data filter does: the reader is
+ * offered them, not obliged to honor them, so the plan keeps the exact check. What it costs is
+ * evaluating the conjunct a second time for the rows the scan emits. What it buys is a reader
+ * free to give up on a file with no page index, or on a row group whose survivors scatter too far
+ * to hold their row ranges, without the answer depending on it.
+ *
+ * Two of the conditions are per scan, and failing either offers nothing:
+ * - The storage-filter pushdown SQL conf is on.
+ * - [[FileFormat.supportBatch]] holds for the schema the reader will see,
+ * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives
+ * its own vectorized-read decision from. Late materialization needs a batch read, so this asks
+ * about batch support rather than naming a format.
+ *
+ * The rest are per conjunct:
+ * - It is deterministic. A reader evaluates the predicate without
+ * `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a
+ * `Nondeterministic` expression, so a non-deterministic conjunct would fail at task time.
+ * - It references at least one column, and every column it references is a projected data
+ * column. A reference to something the scan does not read cannot be evaluated by the reader.
+ * - [[FileFormat.supportsStorageFilter]] accepts it. That is where the expression shapes and
+ * column types a reader can evaluate live, so this method names neither a format nor a type.
+ *
+ * One last condition is on the set that survives: at least one projected data column must be left
+ * for the reader to prune. A scan that projects nothing but the filter's own key columns reads
+ * the same columns for the same rows either way, since the reader has to read a key column to
+ * evaluate the filter on it, so offering it could only add the cost of evaluating the predicate
+ * outside the generated code. Nothing is offered in that case.
+ */
+ private def storageFiltersFor(
+ afterScanFilters: ExpressionSet,
+ fsRelation: HadoopFsRelation,
+ readDataColumns: Seq[Attribute],
+ outputDataSchema: StructType): Seq[Expression] = {
+ val sparkSession = fsRelation.sparkSession
+ val sqlConf = sparkSession.sessionState.conf
+ if (!sqlConf.parquetStorageFilterPushdownEnabled) return Nil
+ val resultSchema = StructType(fsRelation.partitionSchema.fields ++ outputDataSchema.fields)
+ if (!fsRelation.fileFormat.supportBatch(sparkSession, resultSchema)) return Nil
+
+ val dataAttrs = AttributeSet(readDataColumns)
+ val offered = afterScanFilters.filter { expr =>
Review Comment:
`afterScanFilters` is an `ExpressionSet`, and `ExpressionSet.filter` applies the predicate to `e.canonicalized`, not to the original expression. So the new public hook `FileFormat.supportsStorageFilter(expr)` receives a canonicalized expression (attribute names become `none`, metadata is dropped, nullability changes), while `buildReaderWithStorageFilters` receives the original. `ParquetFileFormat` only looks at exprIds and data types, so it is fine today, but a third-party format deciding by column name or field metadata (e.g., column mapping's physical name) would answer wrongly. Could we use `afterScanFilters.toSeq.filter { ... }` or document this contract?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Not applying the storage filter to {}: reading part of a row group needs a "
+ + "Parquet offset index, and this file was written without a page index for at least "
+ + "one projected column", e, MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is computed for a row group whose filter was given up: it read what a plain scan
+ // reads, so the answer is a certain zero. `needBytes`, not just `bytesAvoidedPf != null`,
+ // because that is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null && !filterGivenUp) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the baseline
+ // after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. `getRowRanges` is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then. It has to be
+ // guarded the same way phase 0 guards it, since it consults the pushed filter but not the
+ // conf that turns column-index filtering off.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
Review Comment:
The same "give up" state is signaled through several channels: the boolean return of `appendSurvivorRowToAccumulators` and the local `accumulate` always equal `spliceCurrentRowGroup`; `evaluateStorageFilter` signals the filter give-up both via the `filterGivenUp` field and a `null` return, but the caller never checks for `null`; and `survivorRangeCount` is an implicit out-parameter between methods. Since `(filterGivenUp, spliceCurrentRowGroup)` has only three legal states (splice, filter-only, plain), could we model it as a single enum and let the caller check `survivors == null`?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Not applying the storage filter to {}: reading part of a row group needs a "
+ + "Parquet offset index, and this file was written without a page index for at least "
+ + "one projected column", e, MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is computed for a row group whose filter was given up: it read what a plain scan
+ // reads, so the answer is a certain zero. `needBytes`, not just `bytesAvoidedPf != null`,
+ // because that is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null && !filterGivenUp) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the baseline
+ // after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. `getRowRanges` is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then. It has to be
+ // guarded the same way phase 0 guards it, since it consults the pushed filter but not the
+ // conf that turns column-index filtering off.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
+ // The ranges being built are about to be thrown away, so stop evaluating the rest.
+ if (filterGivenUp) return null;
+ }
+ }
+ }
+ remaining -= num;
+ }
+
+ if (accumulate) {
+ finalizePartialAccumulators();
+ }
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ // Assigned before the loop on purpose: an allocation failure part way through then leaves the
+ // vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows
+ * each.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, pushing them onto their
+ * queues once full. All key columns advance in lockstep, which is what keeps the queues aligned.
+ *
+ * <p>Returns false when the buffered survivors have passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} and this row group has
+ * given splicing up, in which case it has already released what it held and the caller must stop
+ * calling this.
+ */
+ private boolean appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ long valueBytes = 0L;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ // Measured on the destination: a dictionary-encoded source has no length of its own, since
+ // its values are read through the dictionary.
+ if (keyVariableLength[i]) valueBytes += dst.getArrayLength(dstRow);
+ }
+ }
+ splicedBytes += keyFixedBytesPerRow + valueBytes;
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ // Checked only here, so the cost is one comparison per capacity-sized vector rather than per
+ // row. A row group whose survivors fit in a single accumulator is never checked at all: it
+ // then holds one capacity-sized vector per key column, which is what a plain read holds.
+ //
+ // Two allocations share the budget, and giving splicing up releases only the first, so the
+ // cheaper concession comes first: drop the buffer, and give the filter up as well if the row
+ // ranges alone still do not fit. The second measurement is taken after that concession, so it
+ // counts the leaves phase 2 will now drive, which is every projected column rather than the
+ // non-key ones. Either way this row group stops accumulating.
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ if (splicedBytes + rowRangeStateBytes(survivorRangeCount) > cap) {
+ abandonSplicing();
+ // Splicing is already gone, so the flag is all that is left to set.
+ if (rowRangeStateBytes(survivorRangeCount) > cap) filterGivenUp = true;
+ return false;
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+ }
+ return true;
+ }
+
+ /**
+ * Gives the filter up for the row group being read, which leaves phase 2 reading every projected
+ * column over the rows the pushed data filter allowed, exactly what a plain read does. Splicing
+ * goes with it: the buffered survivors are no longer the rows that will be emitted.
+ */
+ private void giveUpFilter() {
+ filterGivenUp = true;
+ abandonSplicing();
+ }
+
+ /**
+ * Gives up splicing for the row group being evaluated and releases every survivor vector it has
+ * buffered. Phase 2 then reads the full projected schema and the emit path takes the persistent
+ * batch, so the rows are unaffected.
+ */
+ private void abandonSplicing() {
+ for (java.util.ArrayDeque<WritableColumnVector> q : keyVectorQueues) {
+ for (WritableColumnVector v : q) v.close();
+ q.clear();
+ }
+ closeAll(currentKeyAccumulators);
+ Arrays.fill(currentKeyAccumulators, null);
+ currentKeyAccumulatorRowCount = 0;
+ spliceCurrentRowGroup = false;
+ }
+
+ /**
+ * Pushes any partially-filled accumulator into its queue at row-group end so the emit path can
+ * dequeue it as the row group's final batch.
+ */
+ private void finalizePartialAccumulators() {
+ if (currentKeyAccumulatorRowCount == 0) return;
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ keyVectorQueues[i].addLast(currentKeyAccumulators[i]);
+ currentKeyAccumulators[i] = null;
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Closes anything held by the splicing path: every vector still queued, the published head
+ * included, and partially-filled accumulators. Called from {@link #close()}.
+ */
+ private void closeSplicingState() {
Review Comment:
`closeSplicingState` re-implements almost the same body as `abandonSplicing`, and the push-to-queue loop is duplicated between `appendSurvivorRowToAccumulators` and `finalizePartialAccumulators`. Similarly, phase 0's `useColumnIndexFilter ? getRowRanges(...) : createSingle(...)` logic with the empty-block/zero-row guards is copied into `recordFileSkipped`. Since these handle off-heap vector lifetimes and the metric baseline, changing only one copy could lead to double-free/leak or inconsistent metrics. Could `closeSplicingState` call `abandonSplicing()`, and could the range computation be one helper such as `pushedFilterRanges(blockIdx)`?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, ExprUtils, Literal, Predicate, XxHash64}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.{BinaryType, BooleanType, ByteType, DataType, DateType, DayTimeIntervalType, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType, StringType, StructType, TimestampNTZType, TimestampType, TimeType, YearMonthIntervalType}
+
+/**
+ * Optional SQL metrics the reader updates while applying a [[ParquetStorageFilter]]. Every
+ * counter is scoped to what the storage filter added on top of a read of the same projection
+ * without one. All fields are nullable; a null field disables that metric.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums the rows those skips excluded, per skipped block the rows that
+ * survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it, and it is zero on an
+ * all-keys projection, which can avoid nothing.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned.
+ *
+ * The row counters' suffix says where a row was excluded, not what would have saved it: a row
+ * inside a kept row group is read as part of its page and dropped during decode, so page
+ * filtering did not save it. An all-keys projection has no page filtering at all, and
+ * [[rowsExcludedWithinRowGroup]] still counts every row the filter dropped.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric = null,
+ rowsExcludedByRowGroup: SQLMetric = null,
+ rowsExcludedWithinRowGroup: SQLMetric = null,
+ bytesAvoidedByRowGroup: SQLMetric = null,
+ bytesAvoidedByPageFiltering: SQLMetric = null)
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[boundExpression]] has its references rewritten to
+ * [[BoundReference]]s pointing at positions 0..(keyColumnIndices.length - 1); the reader must
+ * evaluate it against rows whose fields correspond to those key columns in that order.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ val boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer
+ // construction to first use on the executor.
+ @transient private lazy val predicate: BasePredicate = Predicate.create(boundExpression)
+
+ // Rewrites are cached for the task, because `rewriteForMissingKeys` rebuilds the expression with
+ // `transform`, and a rebuilt `BloomFilterMightContain` has to deserialize its filter again, up to
+ // megabytes per file otherwise. The missing positions are the whole key: the values a caller
+ // substitutes for them are the ones the scan's schema defines, so every file of that scan passes
+ // the same ones.
+ @transient private lazy val rewrites =
+ new ConcurrentHashMap[Seq[Int], ParquetStorageFilter]()
+
+ def test(keyRow: InternalRow): Boolean = predicate.eval(keyRow)
+
+ /**
+ * Returns a new filter for a physical file that is missing some key columns (schema evolution).
+ * The [[BoundReference]]s at `missingKeyLocalPositions`, which are indices into
+ * [[keyColumnIndices]], are replaced by `missingKeyValues`, and the remaining references are
+ * renumbered onto the reduced key-row layout. [[keyColumnIndices]] keeps the present columns in
+ * their original relative order, and SQL metrics are shared with `this`.
+ *
+ * `missingKeyValues(i)` must be the internal-format value the reader produces for a missing
+ * column: its existence DEFAULT when it has one, else null. The result is cached per set of
+ * missing positions, so those values have to be a function of the positions, which they are: they
+ * come from the scan's schema. `ParquetColumnVector` writes that
+ * default into the output vector, so substituting null instead would filter on a value the scan
+ * never returns and could drop matching rows.
+ *
+ * The predicate has to be evaluated against the substitution rather than skipped, because a null
+ * key does not always mean `false`: a `Coalesce`-wrapped reference still produces a non-null
+ * result, and `XxHash64` is `nullable = false` and hashes a null input to its seed.
+ *
+ * With every key position missing, the result holds no [[BoundReference]] at all and
+ * [[evalAllMissing]] can read off its constant truth value.
+ */
+ def rewriteForMissingKeys(
+ missingKeyLocalPositions: Array[Int],
+ missingKeyValues: Array[Any]): ParquetStorageFilter = {
+ require(missingKeyLocalPositions.length == missingKeyValues.length,
+ "missingKeyLocalPositions and missingKeyValues must have the same length")
+ rewrites.computeIfAbsent(missingKeyLocalPositions.toSeq,
+ _ => rewrite(missingKeyLocalPositions, missingKeyValues))
+ }
+
+ private def rewrite(
+ missingKeyLocalPositions: Array[Int],
+ missingKeyValues: Array[Any]): ParquetStorageFilter = {
+ val substitution = missingKeyLocalPositions.zip(missingKeyValues).toMap
+ val presentPositions = keyColumnIndices.indices.filterNot(substitution.contains)
+ val newPosOf = presentPositions.zipWithIndex.toMap
+ val rewritten = boundExpression.transform {
+ case b: BoundReference if substitution.contains(b.ordinal) =>
+ Literal(substitution(b.ordinal), b.dataType)
+ case b: BoundReference => BoundReference(newPosOf(b.ordinal), b.dataType, b.nullable)
+ }
+ val newKeyColumnIndices = presentPositions.map(keyColumnIndices(_)).toArray
+ new ParquetStorageFilter(newKeyColumnIndices, rewritten, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * When every key column is missing (i.e. [[keyColumnIndices]] is empty after
+ * [[rewriteForMissingKeys]]), the bound expression is fully constant. Evaluates it and returns
+ * `true` iff the predicate is literally true; a null or false result is interpreted as "drop
+ * every row" by the reader.
+ */
+ def evalAllMissing(): Boolean = {
+ require(keyColumnIndices.isEmpty, "evalAllMissing only valid when all key columns are missing")
+ boundExpression.eval(InternalRow.empty) == true
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `extractStorageFilters` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = StorageFilterMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted.
+ //
+ // `sorted` is load-bearing. Both `keyColumnIndices` and the remapped references derive from
+ // this list, so any order would keep those two consistent, but the reader's emit path does
+ // not go through the remapping: it pairs the k-th key slot of the output batch with key-row
+ // position k, which is the identity only while this list is ascending.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val unsupported = originalOrdinals.map(requestedSchema.fields(_))
+ .filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type. This is the single authority on
+ * that: [[isSupportedStorageFilter]] consults it for what the planner asks, and [[create]]
+ * re-checks it, so the reader's per-type value copier
+ * (`VectorizedParquetRecordReader.copierFor`) is only ever asked for a type listed here. Adding
+ * a type here without teaching `copierFor` about it turns a planning-time rejection into a task
+ * failure.
+ *
+ * Narrower than `AtomicType`, for two different reasons:
+ * - `VariantType` cannot be supported: its Parquet representation is a group, not a primitive
+ * leaf, so phase 1 has nothing flat to read it into. (It is unreachable anyway, since
+ * `HashExpression.checkInputDataTypes` rejects variant, so no bloom can be built on one.)
+ * - `GeometryType` and `GeographyType` could be supported. Both map to a primitive Parquet
+ * BINARY and both are handled by `WritableColumnVector.isArray`, so the existing byte-array
+ * copier would work. But no bloom can currently reference them: `HashExpression`'s codegen
+ * type dispatch has no case for either, so hashing one fails at codegen. They are left out
+ * until something can actually produce such a filter.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = dt match {
+ case _: BooleanType | _: ByteType | _: ShortType | _: IntegerType | _: LongType => true
+ case _: FloatType | _: DoubleType | _: DecimalType => true
+ case _: DateType | _: TimestampType | _: TimestampNTZType | _: TimeType => true
+ case _: YearMonthIntervalType | _: DayTimeIntervalType => true
+ // StringType also covers CharType and VarcharType, which extend it.
+ case _: StringType | _: BinaryType => true
+ case _ => false
+ }
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner.
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ // The whole conjunct has to be the bloom, not something with a bloom nested under an OR or a
+ // NOT: the reader evaluates the expression it is given and treats a false as "drop this row".
+ // Every reference is checked, not just the ones on the value side, because [[create]] binds
+ // and type-checks all of them.
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) && canEvaluateOnEveryRow(bloom)
+ case _ => false
+ }
+
+ /**
+ * Whether the reader may evaluate `bloom`'s value side on any row of the files it reads.
+ *
+ * It has to ask, because the reader evaluates the predicate on every row the pushed data filter
+ * left, while in the plan the conjunct ran after the ones ahead of it and was skipped for the
+ * rows they rejected. A `CAST(s AS BIGINT)` key, which `InjectRuntimeFilter` builds for a
+ * string-to-long join, then throws in ANSI mode on a row an earlier conjunct would have dropped,
+ * a query that succeeds without this feature.
+ *
+ * So the value side must be a hash of expressions that cannot fail on any input, which
+ * [[ExprUtils.canEvaluateUnconditionally]] decides. `XxHash64` and the membership test itself are
+ * total for every type they accept at analysis time.
+ *
+ * This is a question about the conjunct as the planner holds it, with attribute references for
+ * leaves. It is not one to ask of a bound expression, which that whitelist does not admit.
+ */
+ private def canEvaluateOnEveryRow(bloom: BloomFilterMightContain): Boolean =
+ bloom.valueExpression match {
+ case hash: XxHash64 => hash.children.forall(ExprUtils.canEvaluateUnconditionally)
Review Comment:
`ExprUtils.canEvaluateUnconditionally` has no `Cast` in its whitelist, so every cast key is rejected, including widening casts that can never fail. For example, for an `int` column joined with a `bigint` column, type coercion inserts `CAST(int_col AS BIGINT)` and `InjectRuntimeFilter` uses it as the `XxHash64` child, so this common runtime filter is never pushed when the application side is the `int` table.
The root cause is that the reader evaluates the predicate on rows an earlier conjunct would have rejected. Since the reader is already free to give the filter up, could it instead keep the row (or call `giveUpFilter()` for the row group) on an evaluation error, so that the post-scan `Filter` reproduces the original order and error semantics? At least, could we allow upcasts that cannot fail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-5794832754
Thanks @dongjoon-hyun for the review, I will adjust this PR and answer your findings soon, but let me move this to draft for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4084285791
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -421,11 +552,313 @@ public boolean nextBatch() throws IOException {
return true;
}
+ /**
+ * Splicing emit path. Closes the previous emit's dequeued key vectors (releasing survivor memory
+ * incrementally), advances to the next row group if needed via {@link #checkEndOfRowGroup()},
+ * dequeues one survivor key vector per key column, drives non-key column readers for {@code num}
+ * rows, and assembles a fresh {@link ColumnarBatch} interleaving key (dequeued) and non-key
+ * (persistent value-vector) slots in the original projection order.
+ * The per-emit batch is a transient view over vectors owned elsewhere; see {@link #close()} and
+ * {@link #closeSplicingState()}.
+ */
+ private boolean nextBatchSplicing() throws IOException {
+ if (pendingCloseKeyVectors != null) {
+ for (WritableColumnVector v : pendingCloseKeyVectors) {
+ if (v != null) v.close();
+ }
+ pendingCloseKeyVectors = null;
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) continue;
+ columnVectors[i].reset();
+ }
+ // Match the eager path and zero the outgoing batch before the terminal checks below. Without
+ // this, a terminal call leaves `columnarBatch` pointing at the previous emit whose key slots
+ // were just closed above -- with off-heap vectors those buffers are already freed, so a
+ // consumer that read the batch after nextBatch() returned false would see freed memory.
+ if (columnarBatch != null) columnarBatch.setNumRows(0);
+ if (hitEndOfData) return false;
+ if (rowsReturned >= totalRowCount) return false;
+ checkEndOfRowGroup();
+ if (hitEndOfData) return false;
+
+ int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned);
+
+ WritableColumnVector[] dequeued = new WritableColumnVector[keyVectorQueues.length];
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ dequeued[i] = keyVectorQueues[i].removeFirst();
+ }
+
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) continue;
+ ParquetColumnVector cv = columnVectors[i];
+ for (ParquetColumnVector leafCv : cv.getLeaves()) {
+ VectorizedColumnReader columnReader = leafCv.getColumnReader();
+ if (columnReader != null) {
+ columnReader.readBatch(num, leafCv.getValueVector(),
+ leafCv.getRepetitionLevelVector(), leafCv.getDefinitionLevelVector());
+ }
+ }
+ cv.assemble();
+ }
+ if (rowIndexGenerator != null) {
+ // Row-index column is identified by name (ROW_INDEX_TEMPORARY_COLUMN_NAME), which is a
+ // synthetic metadata column never referenced by a storage filter, so its slot is a non-key
+ // slot with a persistent ParquetColumnVector.
+ rowIndexGenerator.populateRowIndex(columnVectors, num);
+ }
+
+ ColumnVector[] cols = new ColumnVector[persistentBatchColumns.length];
+ // This walks batch slots in ascending order while `keyIdx` walks the survivor queues in
+ // key-row-position order, so it pairs the k-th smallest key slot with key-row position k.
+ // That is only the identity because `ParquetStorageFilter.create` sorts `keyColumnIndices`
+ // ascending -- see the comment there. `isKeyTopLevel` marks which slots are keys but not their
+ // position in that list, so this loop cannot reconstruct the pairing on its own: if the list
+ // ever stops being sorted, key columns silently swap places in the output batch.
+ int keyIdx = 0;
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ if (i < isKeyTopLevel.length && isKeyTopLevel[i]) {
+ cols[i] = dequeued[keyIdx++];
+ } else {
+ cols[i] = persistentBatchColumns[i];
+ }
+ }
+ columnarBatch = new ColumnarBatch(cols);
+ columnarBatch.setNumRows(num);
+
+ rowsReturned += num;
+ numBatched = num;
+ batchIdx = 0;
+ pendingCloseKeyVectors = dequeued;
+ return true;
+ }
+
private void initializeInternal() throws IOException, UnsupportedOperationException {
missingColumns = new HashSet<>();
for (ParquetColumn column : CollectionConverters.asJava(parquetColumn.children())) {
checkColumn(column);
}
+ if (storageFilter != null) {
+ initializeLateMaterialization();
+ }
+ }
+
+ /**
+ * Sets the storage filter for late materialization. Must be called before {@link #initialize};
+ * {@link #initializeLateMaterialization()} (run from {@link #initialize}) inspects the per-file
+ * schema and decides whether splicing actually engages.
+ *
+ * <p>Splicing does NOT engage in one real case: all key columns are missing from this physical
+ * file under schema evolution. The predicate is rewritten with each missing key replaced by the
+ * constant the reader materializes for it (its existence DEFAULT, else null -- see
+ * {@code ParquetStorageFilter.rewriteForMissingKeys}) and evaluated as a constant: a true result
+ * keeps the file with no filtering, a false/null result skips it entirely. Either way the rows
+ * the scan returns are exactly the rows that satisfy the filter, so this is safe.
+ *
+ * <p>Every OTHER precondition is guaranteed by planning-time checks in
+ * {@code FileSourceStrategy.extractStorageFilters} and {@code ParquetStorageFilter.create}, and
+ * violations throw rather than fall back: once extraction has moved a bloom filter onto the scan
+ * it is gone from the post-scan Filter, so quietly not applying it would produce wrong rows.
+ */
+ public void setStorageFilter(ParquetStorageFilter storageFilter) {
+ this.storageFilter = storageFilter;
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ lateMatReader = reader.getUnderlyingReader();
+ if (lateMatReader == null) {
+ // Unreachable in production: the only ParquetRowGroupReader ParquetFileFormat builds is
+ // ParquetRowGroupReaderImpl, which returns its reader. Dropping the filter here would return
+ // rows it rejects, since extraction already removed it from the post-scan Filter.
+ throw new IllegalStateException(
+ "Storage-filter pushdown requires a reader backed by a ParquetFileReader, but "
+ + reader.getClass().getName() + " does not expose one");
+ }
+ if (configuration != null) {
+ useColumnIndexFilter = configuration.getBoolean(
+ ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ }
+
+ // Resolve each key column's top-level ParquetColumn. Partition into present and missing (the
+ // latter can happen under schema evolution: a column is in the requested schema but not in this
+ // physical parquet file). For any non-primitive key we still bail; phase-1 reads only primitive
+ // leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> missingKeyLocalPositions = new ArrayList<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals. Fail loudly
+ // rather than dropping the filter -- extractStorageFilters already removed it from the
+ // post-scan Filter, so silently ignoring it here would return wrong rows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create. Fail loudly for
+ // the same reason as above.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (missingColumns.contains(column)) {
+ missingKeyLocalPositions.add(i);
+ } else {
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // If any key column is missing from this file, rewrite the predicate to substitute the constant
+ // the reader will actually materialize for that column. That is the column's existence DEFAULT
+ // when it has one (ParquetColumnVector writes it into the output vector and marks the vector
+ // constant), otherwise null. Substituting null for a column that reads back as its default
+ // would filter on a value the scan never returns. The predicate must be evaluated against the
+ // substituted constant rather than skipped: null does not always mean false in a filter.
+ if (!missingKeyLocalPositions.isEmpty()) {
+ int[] missing = new int[missingKeyLocalPositions.size()];
+ Object[] missingValues = new Object[missingKeyLocalPositions.size()];
+ Object[] existenceDefaults =
+ ResolveDefaultColumns.existenceDefaultValues(sparkRequestedSchema);
+ for (int i = 0; i < missing.length; i++) {
+ missing[i] = missingKeyLocalPositions.get(i);
+ missingValues[i] = existenceDefaults[keyIndices[missing[i]]];
+ }
+ storageFilter = storageFilter.rewriteForMissingKeys(missing, missingValues);
+
+ if (presentKeyColumns.isEmpty()) {
+ // All key columns missing: the rewritten predicate is fully constant. Evaluate once and
+ // apply uniformly to the whole file.
+ boolean keepAll = storageFilter.evalAllMissing();
+ if (keepAll) {
+ // Predicate is constant-true for this file: no filtering to do.
+ storageFilter = null;
+ } else {
+ // Predicate is constant-false/null: no row from this file can pass the filter.
+ storageFilter = null;
+ hitEndOfData = true;
+ }
+ return;
+ }
+ }
+
+ keyDescriptors = new ColumnDescriptor[presentKeyColumns.size()];
+ keyRequired = new boolean[presentKeyColumns.size()];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < presentKeyColumns.size(); i++) {
+ ParquetColumn column = presentKeyColumns.get(i);
+ keyDescriptors[i] = column.descriptor().get();
+ keyRequired[i] = column.required();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyRequestedSchema = keySchemaBuilder.named(requestedSchema.getName());
+
+ // Build the non-key (complement) schema. When the projection has at least one non-key column,
+ // phase 2 will switch lateMatReader's schema to this and read those columns under finalRanges.
+ // When the projection is *all* key columns (e.g. a scan whose only column is the bloom probe),
+ // splicing still pays off: phase 2 has nothing useful to read, so we skip it entirely (no read,
+ // no IO) and emit batches purely from the key queues. The `nonKeyRequestedSchema != null` check
+ // downstream gates phase-2 IO.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount > 0) {
+ nonKeyRequestedSchema = nonKeyBuilder.named(requestedSchema.getName());
+ requireOffsetIndexesForPhase2();
+ }
+ initializeSplicingState(presentKeyColumns);
+
+ totalBlockCount = lateMatReader.getRowGroups().size();
+ nextBlockIndex = 0;
+ }
+
+ /**
+ * Fails now if any projected column of any row group lacks a Parquet offset index.
+ *
+ * <p>Phase 2 reads a strict subset of a row group's rows, which parquet can only do via the
+ * offset index; files written before parquet-mr 1.11, or by writers that omit it, have none. We
+ * cannot
+ * widen phase 2 to the whole block instead, because the key vectors already hold only the
+ * survivors and the batch would misalign -- and we cannot skip the filter either, since
+ * {@code extractStorageFilters} has already removed it from the post-scan Filter.
+ *
+ * <p>Every projected column is checked, not just the non-key ones phase 2 reads, because parquet
+ * builds one column index store per row group and reuses it. Phase 0 asks for the row ranges
+ * under the full requested schema, so {@code ColumnIndexStoreImpl.create} is called with the key
+ * columns in its path set, and it returns its {@code EMPTY} singleton as soon as any one of those
+ * paths has no offset index. {@code ParquetFileReader.getColumnIndexStore} memoizes that store
+ * per block, and {@code EMPTY.getOffsetIndex} throws for *every* column. So a key column with no
+ * offset index kills phase 2 too, with a raw {@code MissingOffsetIndexException} naming some
+ * non-key column and none of the guidance below.
+ *
+ * <p>Checking up front rather than at the first partially-kept row group is deliberate: whether
+ * phase 2 needs the offset index otherwise depends on how selective the filter turns out to be on
+ * this particular file, so the same query would fail or not depending on the data. This is
+ * conservative -- a filter that happens to keep every row of every block would not have needed
+ * the offset index -- but such a filter also saves nothing, so failing loudly loses nothing.
+ *
+ * <p>The check itself is free: {@code getOffsetIndexReference()} is a footer field that
+ * {@link #initialize} has already read.
+ */
+ private void requireOffsetIndexesForPhase2() {
+ Set<ColumnPath> projectedPaths = new HashSet<>();
+ for (ColumnDescriptor column : requestedSchema.getColumns()) {
+ projectedPaths.add(ColumnPath.get(column.getPath()));
+ }
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ for (ColumnChunkMetaData chunk : blocks.get(blockIdx).getColumns()) {
+ if (projectedPaths.contains(chunk.getPath()) && chunk.getOffsetIndexReference() == null) {
Review Comment:
Narrowed rather than turned into a per-file fallback, and it came out a net deletion.
The up-front check is gone. Parquet resolves every requested column's offset index before it reads a byte, so phase 2's read is simply wrapped and `MissingOffsetIndexException` rethrown with the conf to set. That is narrower than the check was -- a row group the filter keeps whole never needs the index, since `readFilteredRowGroup` degrades to a plain read when the ranges cover the block, and one it rejects whole is never read at all -- so a file written without a page index still scans as long as the filter never prunes inside a row group. It is also more complete: an IO error while reading an index surfaces the same way, which a footer walk could not see.
The eager-read fallback does not work in this design: phase 1 has already buffered only the survivors, so a whole-block phase 2 would misalign the batch, and dropping the predicate is not an option because extraction has removed it from the post-scan `Filter`. What would remove the failure class entirely is a parquet-side change -- `readFilteredRowGroup` reading whole chunks while keeping the row ranges for `getRowIndexes()` -- and that is now named as a follow-up in the description.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +929,450 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+
+ // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the
+ // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and
+ // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes
+ // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index
+ // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's
+ // ranges under the full schema, and with it off we do not call getRowRanges at all.
+ lateMatReader.setRequestedSchema(requestedSchema);
+ // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT
+ // options.useColumnIndexFilter(), so calling it unconditionally would keep applying
+ // column-index filtering after a user turned it off. That conf is the documented escape hatch
+ // for files whose column index is wrong, and trusting a wrong column index here would drop
+ // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no
+ // longer holds this predicate.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ if (pushedFilterRanges.rowCount() == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // Both byte metrics are always wired in production (FileSourceScanLike creates all five
+ // whenever storageFilters is non-empty), so this only skips the work on the test-only path
+ // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so
+ // there is nothing here to avoid on the production path.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ long baselineRows = pushedFilterRanges.rowCount();
+ long baselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema,
+ pushedFilterRanges)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyRequestedSchema);
+ long phase1Bytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema,
+ pushedFilterRanges)
+ : 0L;
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite "
+ + pushedFilterRanges.rowCount() + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
+
+ if (finalRanges.rowCount() == 0) {
+ // Every surviving row was rejected by the storage filter; skip block entirely. We still
+ // paid phase-1 to read the key column, so the bytes avoided vs a no-storage-filter read
+ // are baseline - phase1 (the non-key bytes the no-filter path would have read).
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup();
+ if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows);
+ if (bytesAvoidedRg != null) bytesAvoidedRg.add(baselineBytes - phase1Bytes);
+ continue;
+ }
+
+ // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely
+ // when the projection is all keys (nonKeyRequestedSchema is null); emit reconstructs each
+ // batch from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyRequestedSchema == null) {
+ keptRows = finalRanges.rowCount();
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(nonKeyRequestedSchema);
+ // requireOffsetIndexesForPhase2() already established that every projected column of every
+ // row group has an offset index, so this page-filtering read cannot fail for want of one.
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRanges.rowCount()
+ + " surviving rows");
+ }
+ keptRows = dataPages.getRowCount();
+ phase2Bytes = bytesAvoidedPf != null
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, nonKeyRequestedSchema,
+ finalRanges)
+ : 0L;
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null) {
+ bytesAvoidedPf.add(baselineBytes - phase1Bytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the leaf columns of {@code schema} when it reads
+ * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are
+ * included, because both are read whenever any page of a chunk is read.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters --
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them. Every caller for a given block walks the same
+ * metadata, so a skipped column drops out of the baseline and the per-phase totals alike.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ MessageType schema,
+ RowRanges rowRanges) {
+ if (schema == null || rowRanges.rowCount() == 0 || schema.getColumns().isEmpty()) {
+ return 0L;
+ }
+ BlockMetaData block = reader.getRowGroups().get(blockIndex);
+ long blockRowCount = block.getRowCount();
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : block.getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ boolean wholeBlock = rowRanges.rowCount() == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : schema.getColumns()) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Reads all rows of the given key-only {@link PageReadStore} (which contains only rows in
+ * {@code pushedFilterRanges}) in capacity-sized chunks, evaluates the storage filter on each row,
+ * builds a {@link RowRanges} of surviving rows in original block-row coordinates, and appends
+ * survivor key values into the per-key-column accumulators ({@link #currentKeyAccumulators}).
+ * When an accumulator hits {@link #capacity}, it's pushed into {@link #keyVectorQueues} and a
+ * fresh one is allocated. After all rows have been examined, any partial trailing accumulator is
+ * pushed too.
+ *
+ * <p>The result is a subset of {@code pushedFilterRanges}: rows not in {@code
+ * pushedFilterRanges} were never read and are implicitly excluded.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ long keyRowsTotal = pushedFilterRanges.rowCount();
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
Review Comment:
Bounded now, in the stopgap form you suggested, and the conf is `spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes` (internal, 64MB). Phase 1 counts the bytes it has buffered and past the cap releases them and gives that row group up: phase 2 then reads every projected column of the surviving rows, so nothing is buffered and the cost is one extra read of the key columns. The count is examined once per accumulator, so the hot loop pays one add per key column and the check costs one comparison per batch worth of survivors.
Worth recording why it counts rather than estimates, since the first version did estimate from the footer. `ColumnChunkMetaData.getTotalUncompressedSize()` is the size of the uncompressed *pages*, which still hold dictionary ids: measured, a 2M-row BIGINT column with 1000 distinct values reports 1.26 bytes/row dictionary-encoded against the 9 the accumulator holds. Spark writes dictionary-encoded by default, so that estimate underestimated by 7x on the default path.
Chunk-at-a-time is discussed in the description. It would shorten what the first batch waits for, but the IO of a `LIMIT` query is unchanged either way, and chunking turns phase 2 into one read call per chunk and refetches any page that straddles a boundary.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +929,450 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}.
+ * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection).
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+
+ // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the
+ // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and
+ // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes
+ // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index
+ // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's
+ // ranges under the full schema, and with it off we do not call getRowRanges at all.
+ lateMatReader.setRequestedSchema(requestedSchema);
+ // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT
+ // options.useColumnIndexFilter(), so calling it unconditionally would keep applying
+ // column-index filtering after a user turned it off. That conf is the documented escape hatch
+ // for files whose column index is wrong, and trusting a wrong column index here would drop
+ // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no
+ // longer holds this predicate.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ if (pushedFilterRanges.rowCount() == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // Both byte metrics are always wired in production (FileSourceScanLike creates all five
+ // whenever storageFilters is non-empty), so this only skips the work on the test-only path
+ // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so
+ // there is nothing here to avoid on the production path.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null;
+ long baselineRows = pushedFilterRanges.rowCount();
+ long baselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema,
+ pushedFilterRanges)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row.
+ lateMatReader.setRequestedSchema(keyOnlyRequestedSchema);
+ long phase1Bytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema,
+ pushedFilterRanges)
+ : 0L;
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop its
+ // surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite "
+ + pushedFilterRanges.rowCount() + " rows selected by the pushed filter");
+ }
+ RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges);
Review Comment:
It is a latency and CPU cost, not an IO one: `internalReadFilteredRowGroup` reads all of the row group's requested chunks before it hands back a `PageReadStore`, so `LIMIT 5` transfers the same bytes on the eager path too. What the first batch waits for is one row group's key decode and predicate evaluation, and the buffer cap now bounds that as well.
Chunk-at-a-time would shorten it, but at a price the description spells out: one read call per chunk instead of one per row group, pages that straddle a chunk boundary fetched more than once with nothing caching them (parquet caches footers, not pages, and S3A's default input stream caches nothing either), and the coalescing and parallelism of a single vectored read given up -- parquet 1.18 has vectored IO on by default. So it is not filed.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,1965 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayOutputStream, File}
+import java.net.URI
+import java.time.LocalTime
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapreduce.Job
+import org.apache.parquet.hadoop.{ParquetInputFormat, ParquetOutputFormat}
+
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, BloomFilterMightContain, BoundReference, Coalesce, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, XxHash64}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FileSourceScanLike, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests the late-materialization path of [[VectorizedParquetRecordReader]] driven by a
+ * [[ParquetStorageFilter]]. Writes small multi-row-group parquet files, wires a hand-built filter
+ * into the reader, and asserts correctness + the two storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession {
+ import testImplicits._
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = rows.toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. Note this does NOT disable the
+ // column index, despite what an earlier version of this comment claimed.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Collects all rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) = {
+ val reader = new VectorizedParquetRecordReader(false, 4096)
+ reader.setStorageFilter(storageFilter)
+ reader.initialize(filePath, java.util.Arrays.asList("k", "v"))
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch().asInstanceOf[ColumnarBatch]
+ val n = batch.numRows()
+ val kVec = batch.column(0)
+ val vVec = batch.column(1)
+ var i = 0
+ while (i < n) {
+ collected += ((kVec.getLong(i), vVec.getUTF8String(i).toString))
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
Review Comment:
Fixed in all four: the drain happens inside `Utils.tryInitializeResource`, which closes the reader if anything in the loop throws and leaves it open otherwise, so the caller's `finally` still owns the close on the passing path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4103839051
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
Review Comment:
**Non-blocking (P2):** Please close this phase-1 `PageReadStore` after `evaluateStorageFilter` finishes, including skip, give-up, and exception exits. `readFilteredRowGroup` returns an `AutoCloseable` store whose page readers and `ByteBufferReleaser` otherwise stay live; this path acquires one extra store for every attempted row group, so long scans can accumulate direct-memory pressure. A try-with-resources around the phase-1 acquisition and evaluation matches the ownership boundary.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Not applying the storage filter to {}: reading part of a row group needs a "
+ + "Parquet offset index, and this file was written without a page index for at least "
+ + "one projected column", e, MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is computed for a row group whose filter was given up: it read what a plain scan
+ // reads, so the answer is a certain zero. `needBytes`, not just `bytesAvoidedPf != null`,
+ // because that is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null && !filterGivenUp) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the baseline
+ // after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. `getRowRanges` is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then. It has to be
+ // guarded the same way phase 0 guards it, since it consults the pushed filter but not the
+ // conf that turns column-index filtering off.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
Review Comment:
**Blocking (P1):** Once this returns false, `evaluateStorageFilter` still adds every later match to `finalRangesBuilder`, but the per-survivor budget check is disabled with accumulation and the remaining check happens only after the full row group. Wide keys can therefore give up splicing early and still build an unbounded scattered range list, violating the configured per-row-group memory cap. Please make the survivor transition own both key-byte and range accounting, including after splicing is abandoned, and fall back as soon as the combined budget is crossed.
**Recommended change:** Centralize the combined spliced-value and row-range budget check in the per-survivor loop, apply it to partial accumulators and filter-only mode, and stop retaining survivor state immediately when the bound is exceeded. Align the config documentation and add boundary scenarios for large values and post-give-up scattered ranges.
**Why this works:** After each survivor changes retained value or range state, compute the same combined accounting regardless of accumulator fullness or spliceCurrentRowGroup. If the next retained state exceeds the cap, release key accumulators, abandon the storage filter for that row group, and avoid constructing the remainder of the survivor range list.
**Scope:** Make one row-group survivor-state owner enforce the advertised memory bound in every accumulation mode.
**Compatibility:** Eligible scans may still splice within budget, while every give-up path continues to behave as an optional optimization and preserves query results.
**Risks:** Giving up in the middle of evaluation must not leave partial key vectors or ranges observable by emit. Checking before versus after adding a survivor must use a consistent inclusive cap boundary.
**Constraints:** Fallback must keep every row selected by pushedFilterRanges because the post-scan conjunct remains authoritative. The accounting may acknowledge column-vector capacity slack, but it must not claim a hard bound it does not enforce.
**Success:** A partial accumulator of large variable-width keys cannot remain retained beyond the configured combined budget. Once splicing is abandoned, later scattered survivors cannot grow range state beyond the same budget. Budget give-up returns the same rows as a plain read and releases partial survivor vectors.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
Review Comment:
**Blocking (P1):** This key-only read is the first offset-index-dependent operation, but it sits outside the `MissingOffsetIndexException` fallback used for phase 2. A valid mixed-index file can have `pushedFilterRanges` narrowed by one indexed column while this key column lacks an offset index; enabling this optional optimization then fails a query the plain reader accepts. Please extend or preflight the fallback at this phase-1 boundary so the row group is read plainly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4106889093
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
Review Comment:
You are right, and the reasoning is exactly right too: the key the post-scan `Filter` sees is the correct one, so it cannot catch a non-key value taken from the wrong row.
Fixed by making the conf turn the whole path off rather than phase 0 alone. With `parquet.filter.columnindex.enabled=false` the reader declines the storage filter at init, so nothing reads part of a row group. The escape hatch is then total, which is what someone setting it means.
It also removes code: the phase-0 branch and the helper that held it are gone, and both sites now ask parquet for the ranges directly. The conf doc says the feature is off in that case. The test asserts the scan hands back every row of the file and reports all five metrics as zero, and that the same read with the conf on reports a saving, so it cannot pass by the feature being off everywhere. 2d8d39b
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Not applying the storage filter to {}: reading part of a row group needs a "
+ + "Parquet offset index, and this file was written without a page index for at least "
+ + "one projected column", e, MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is computed for a row group whose filter was given up: it read what a plain scan
+ // reads, so the answer is a certain zero. `needBytes`, not just `bytesAvoidedPf != null`,
+ // because that is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null && !filterGivenUp) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the baseline
+ // after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. `getRowRanges` is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then. It has to be
+ // guarded the same way phase 0 guards it, since it consults the pushed filter but not the
+ // conf that turns column-index filtering off.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
+ // The ranges being built are about to be thrown away, so stop evaluating the rest.
+ if (filterGivenUp) return null;
+ }
+ }
+ }
+ remaining -= num;
+ }
+
+ if (accumulate) {
+ finalizePartialAccumulators();
+ }
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ // Assigned before the loop on purpose: an allocation failure part way through then leaves the
+ // vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyDescriptors.length];
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < keyDescriptors.length; i++) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ keyScratchVectors[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ keyScratchBatch = new ColumnarBatch(keyScratchVectors);
+ }
+
+ /**
+ * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows
+ * each.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP;
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType();
+ currentKeyAccumulators[i] = useOffHeap
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Appends row {@code srcRow} of every key column to the accumulators, pushing them onto their
+ * queues once full. All key columns advance in lockstep, which is what keeps the queues aligned.
+ *
+ * <p>Returns false when the buffered survivors have passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} and this row group has
+ * given splicing up, in which case it has already released what it held and the caller must stop
+ * calling this.
+ */
+ private boolean appendSurvivorRowToAccumulators(int srcRow) {
+ final int dstRow = currentKeyAccumulatorRowCount;
+ final WritableColumnVector[] accs = currentKeyAccumulators;
+ final WritableColumnVector[] srcs = keyScratchVectors;
+ final ValueCopier[] copiers = keyCopiers;
+ long valueBytes = 0L;
+ for (int i = 0, n = accs.length; i < n; i++) {
+ WritableColumnVector src = srcs[i];
+ WritableColumnVector dst = accs[i];
+ if (src.isNullAt(srcRow)) {
+ dst.putNull(dstRow);
+ } else {
+ copiers[i].copy(dst, dstRow, src, srcRow);
+ // Measured on the destination: a dictionary-encoded source has no length of its own, since
+ // its values are read through the dictionary.
+ if (keyVariableLength[i]) valueBytes += dst.getArrayLength(dstRow);
+ }
+ }
+ splicedBytes += keyFixedBytesPerRow + valueBytes;
+ currentKeyAccumulatorRowCount = dstRow + 1;
+ if (currentKeyAccumulatorRowCount == capacity) {
Review Comment:
Both real, and one owner weighs both halves now. The check runs after every surviving row, over `splicedBytes + rowRangeStateBytes(survivorRangeCount)`, and there is no second check anywhere else. So a row group whose survivors fit in a single accumulator is weighed like any other, and crossing the cap cannot be an accumulator late.
Crossing it takes the cheaper concession first: release the buffer, and give the filter up as well only if the ranges alone still do not fit.
The doc claim is gone. It now says a task holds up to one extra copy of the key columns for one row group, and that what is counted is the buffered values and their per-row overhead, not the backing arrays a column vector may grow beyond that. @cloud-fan asked for the same on the range half, so this answers both. c29625f
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -409,23 +622,244 @@ public boolean nextBatch() throws IOException {
}
cv.assemble();
}
- // If needed, compute row indexes within a file.
- if (rowIndexGenerator != null) {
- rowIndexGenerator.populateRowIndex(columnVectors, num);
- }
+ }
- rowsReturned += num;
+ /** Publishes {@code num} rows as the current batch. */
+ private void finishBatch(int num) {
columnarBatch.setNumRows(num);
+ rowsReturned += num;
numBatched = num;
batchIdx = 0;
- return true;
+ }
+
+ /**
+ * Points the batch's key slots at one survivor key vector each, which is what publishes them: the
+ * same {@link ColumnarBatch} is handed out every time, over an array it holds by reference. The
+ * queues keep owning those vectors until the next batch releases them, so a phase-2 read that
+ * throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void publishSurvivorKeyVectors(int num) {
+ for (int i = 0; i < keyVectorQueues.length; i++) {
+ if (keyVectorQueues[i].isEmpty()) {
+ // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop
+ // is driven by that same count. Named rather than left to NoSuchElementException.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to "
+ + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num));
+ }
+ }
+ keyVectorsPublished = true;
+ // Key slots are filled in ascending slot order while `keyIdx` walks the queues in key-list
+ // order, so the pairing is the identity only because `ParquetStorageFilter.create` sorts
+ // `keyColumnIndices` ascending. `isKeyTopLevel` says which slots are keys, not where each
+ // sits in that list, so this loop cannot re-derive the pairing: an unsorted list would swap
+ // key columns in the output batch. Only key slots are touched, since a non-key slot never
+ // holds anything but its persistent vector.
+ int keyIdx = 0;
+ for (int i = 0; i < isKeyTopLevel.length; i++) {
+ if (isKeyTopLevel[i]) spliceBatchColumns[i] = keyVectorQueues[keyIdx++].peekFirst();
+ }
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (java.util.ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
}
private void initializeInternal() throws IOException, UnsupportedOperationException {
missingColumns = new HashSet<>();
for (ParquetColumn column : CollectionConverters.asJava(parquetColumn.children())) {
checkColumn(column);
}
+ if (storageFilter != null) {
+ initializeLateMaterialization();
+ }
+ }
+
+ /**
+ * Sets the storage filter for late materialization. Must be called before {@link #initialize};
+ * {@link #initializeLateMaterialization()} then inspects the per-file schema and decides whether
+ * splicing engages.
+ *
+ * <p>It does not engage in one real case: every key column is missing from this physical file
+ * under schema evolution. The predicate is then rewritten with each missing key replaced by the
+ * constant the reader materializes for it (its existence DEFAULT, else null) and evaluated once.
+ * True keeps the file unfiltered, false or null skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by
+ * {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}, so a
+ * violation of it here is a planner bug and is asserted rather than handled. A file the reader
+ * simply cannot prune is a different matter: it reads it the way a plain scan would.
+ */
+ public void setStorageFilter(ParquetStorageFilter storageFilter) {
+ this.storageFilter = storageFilter;
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ lateMatReader = reader.getUnderlyingReader();
+ if (lateMatReader == null) {
+ // Late materialization drives a ParquetFileReader directly, so without one the filter is
+ // simply not applied and the plain read path runs. The conjunct is in the post-scan filter as
+ // well, so the rows it would have dropped are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ useColumnIndexFilter = configuration.getBoolean(
+ ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn. Partition into present and missing (the
+ // latter can happen under schema evolution: a column is in the requested schema but not in this
+ // physical parquet file). For any non-primitive key we still bail; phase-1 reads only primitive
+ // leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> missingKeyLocalPositions = new ArrayList<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (missingColumns.contains(column)) {
+ missingKeyLocalPositions.add(i);
+ } else {
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // If any key column is missing from this file, rewrite the predicate to substitute the constant
+ // the reader will actually materialize for that column. That is the column's existence DEFAULT
+ // when it has one (ParquetColumnVector writes it into the output vector and marks the vector
+ // constant), otherwise null. Substituting null for a column that reads back as its default
+ // would filter on a value the scan never returns. The predicate must be evaluated against the
+ // substituted constant rather than skipped: null does not always mean false in a filter.
+ if (!missingKeyLocalPositions.isEmpty()) {
+ int[] missing = new int[missingKeyLocalPositions.size()];
+ Object[] missingValues = new Object[missingKeyLocalPositions.size()];
+ Object[] existenceDefaults =
+ ResolveDefaultColumns.existenceDefaultValues(sparkRequestedSchema);
+ for (int i = 0; i < missing.length; i++) {
+ missing[i] = missingKeyLocalPositions.get(i);
+ missingValues[i] = existenceDefaults[keyIndices[missing[i]]];
+ }
+ storageFilter = storageFilter.rewriteForMissingKeys(missing, missingValues);
+
+ if (presentKeyColumns.isEmpty()) {
+ // Every key column is missing, so the rewritten predicate is constant for this file.
+ boolean keepAll = storageFilter.evalAllMissing();
+ if (!keepAll) recordFileSkipped();
+ storageFilter = null;
+ hitEndOfData = !keepAll;
+ return;
+ }
+ }
+
+ keyDescriptors = new ColumnDescriptor[presentKeyColumns.size()];
+ keyRequired = new boolean[presentKeyColumns.size()];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < presentKeyColumns.size(); i++) {
+ ParquetColumn column = presentKeyColumns.get(i);
+ keyDescriptors[i] = column.descriptor().get();
+ keyRequired[i] = column.required();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ requestedColumns = requestedSchema.getColumns();
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. A null
+ // `nonKeyColumns` says there is nothing for phase 2 to read, so it is skipped entirely. The
+ // planner does not push a scan of that shape, since such a scan reads what a plain one would,
+ // so this is reachable only by driving the reader directly.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount > 0) {
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ }
+ initializeSplicingState(presentKeyColumns);
+
+ totalBlockCount = lateMatReader.getRowGroups().size();
+ nextBlockIndex = 0;
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues. {@link #initBatch} also reads it, as the sign that this file splices at
+ * all and needs its own batch array.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState(List<ParquetColumn> presentKeyColumns) {
+ int numTop = sparkSchema.fields().length;
+ isKeyTopLevel = new boolean[numTop];
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ for (int slot : keyIndices) {
+ isKeyTopLevel[slot] = true;
+ }
+ int numKeys = presentKeyColumns.size();
+ keyVectorQueues = new java.util.ArrayDeque[numKeys];
+ for (int i = 0; i < numKeys; i++) {
+ keyVectorQueues[i] = new java.util.ArrayDeque<>();
+ }
+ currentKeyAccumulators = new WritableColumnVector[numKeys];
+ currentKeyAccumulatorRowCount = 0;
+ keyCopiers = new ValueCopier[numKeys];
+ keyVariableLength = new boolean[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ DataType dt = fields[keyIndices[i]].dataType();
+ keyCopiers[i] = copierFor(dt);
+ keyVariableLength[i] = isVariableLength(dt);
+ // One null byte per row either way. A fixed-width value adds its own width, a variable-length
+ // one the int offset and int length that point at the byte child.
+ keyFixedBytesPerRow += keyVariableLength[i] ? 1 + 8 : 1 + dt.defaultSize();
+ }
+ }
+
+ /**
+ * What phase 2 of the current row group will hold for its row ranges. Every column reader it
+ * drives materializes the row group's range list of its own ({@code ParquetReadState}), and a
+ * filter whose survivors are scattered makes one range per surviving row.
+ */
+ private long rowRangeStateBytes(long rangeCount) {
Review Comment:
Taken at the root, and thank you for pointing at `constructRanges` rather than at the budget. `ParquetReadState` no longer builds a list at all: it coalesces the row indexes it walks into ranges lazily and holds the current one as two `long`s. The ranges are consumed once, in order, so the list bought nothing. Every read that uses a column index is lighter for it, not only this feature's.
With the `x leaves` factor gone, 64MB is about 1.6M ranges for one row group, and a row group cannot produce that many: the count is bounded by its row count, which at the default `parquet.block.size` is around 1M. So the case you describe stops being reachable, and what the budget still catches is the buffer, which is the half that really does grow with the key width. c29625f
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
Review Comment:
Three of the four are fixed. The footer check is the one I am not taking, and the reason is this PR's own history.
Fixed: phase 1 keeps running on such a file. `fileHasNoOffsetIndex` now only stops the buffering, so the row groups the filter empties are still skipped whole for the rest of the split. The mixed case reads plainly instead of giving the file up, since the filter is given up for that row group and the read retried over `pushedFilterRanges`. The WARN says what happened and what still works. The SQLConf sentence you quote is gone.
The footer walk I did write, and then removed: round 2 of this PR had already deleted the same walk (2cb98b2), for two defects mine had again.
- Over-strict. Parquet resolves the index over the paths current at its first lookup for the block, which with no pushed data filter is phase 2's non-key columns alone. A walk over the whole projection therefore rejects a file whose key column has no offset index, where the read would have succeeded.
- Incomplete. An `IOException` while reading an offset index raises the same exception, and the footer cannot see that.
What it would save is one row group's phase-1 buffering per split and one stack trace, because the throw lands before any data page is read. That is worth less than a check that is wrong in both directions, so the exception stays the source of truth. c29625f
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -771,6 +833,28 @@ case class FileSourceScanExec(
readRDD
}
+ // Materialize scalar subqueries inside storage filters to literals and bind AttributeReferences
+ // to BoundReferences targeting positions in `requiredSchema`. Subqueries must have been prepared
+ // by SparkPlan before this is forced (same contract as `pushedDownFilters`).
+ @transient
+ protected lazy val preparedStorageFilters: Seq[Expression] = {
+ if (storageFilters.isEmpty) {
+ Nil
+ } else {
+ // No conf check here: the conf decides at planning time whether a scan is offered storage
+ // filters at all, and re-reading it now could only make this scan drop work it already has.
+ //
+ // `output` is `readDataColumns ++ generatedMetadataColumns ++ partitionColumns ++
+ // constantMetadataColumns` and `requiredSchema` is the StructType of the first two groups, so
+ // the first `requiredSchema.length` attributes line up with its fields.
+ val requestedDataAttrs = output.take(requiredSchema.length)
+ storageFilters.map { expr =>
+ val subqueryReplaced = expr.transform { case s: execution.ScalarSubquery => s.toLiteral }
Review Comment:
Follow-up, now in the description: broadcast the bytes once and share the deserialized filter per executor, the way `InSubqueryExec.resultBroadcast` does.
The repeated deserialization per task is already down to one. The rewrite for missing key columns is memoized per set of missing positions for the task, so `rewriteForMissingKeys` no longer rebuilds the filter per file. What is left is the second copy in the task binary, and that is inherent to the conjunct staying in the plan, so removing it is the broadcast rather than a local fix. c29625f
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, ExprUtils, Literal, Predicate, XxHash64}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.{BinaryType, BooleanType, ByteType, DataType, DateType, DayTimeIntervalType, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType, StringType, StructType, TimestampNTZType, TimestampType, TimeType, YearMonthIntervalType}
+
+/**
+ * Optional SQL metrics the reader updates while applying a [[ParquetStorageFilter]]. Every
+ * counter is scoped to what the storage filter added on top of a read of the same projection
+ * without one. All fields are nullable; a null field disables that metric.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums the rows those skips excluded, per skipped block the rows that
+ * survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it, and it is zero on an
+ * all-keys projection, which can avoid nothing.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned.
+ *
+ * The row counters' suffix says where a row was excluded, not what would have saved it: a row
+ * inside a kept row group is read as part of its page and dropped during decode, so page
+ * filtering did not save it. An all-keys projection has no page filtering at all, and
+ * [[rowsExcludedWithinRowGroup]] still counts every row the filter dropped.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric = null,
+ rowsExcludedByRowGroup: SQLMetric = null,
+ rowsExcludedWithinRowGroup: SQLMetric = null,
+ bytesAvoidedByRowGroup: SQLMetric = null,
+ bytesAvoidedByPageFiltering: SQLMetric = null)
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[boundExpression]] has its references rewritten to
+ * [[BoundReference]]s pointing at positions 0..(keyColumnIndices.length - 1); the reader must
+ * evaluate it against rows whose fields correspond to those key columns in that order.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ val boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer
+ // construction to first use on the executor.
+ @transient private lazy val predicate: BasePredicate = Predicate.create(boundExpression)
+
+ // Rewrites are cached for the task, because `rewriteForMissingKeys` rebuilds the expression with
+ // `transform`, and a rebuilt `BloomFilterMightContain` has to deserialize its filter again, up to
+ // megabytes per file otherwise. The missing positions are the whole key: the values a caller
+ // substitutes for them are the ones the scan's schema defines, so every file of that scan passes
+ // the same ones.
+ @transient private lazy val rewrites =
+ new ConcurrentHashMap[Seq[Int], ParquetStorageFilter]()
+
+ def test(keyRow: InternalRow): Boolean = predicate.eval(keyRow)
+
+ /**
+ * Returns a new filter for a physical file that is missing some key columns (schema evolution).
+ * The [[BoundReference]]s at `missingKeyLocalPositions`, which are indices into
+ * [[keyColumnIndices]], are replaced by `missingKeyValues`, and the remaining references are
+ * renumbered onto the reduced key-row layout. [[keyColumnIndices]] keeps the present columns in
+ * their original relative order, and SQL metrics are shared with `this`.
+ *
+ * `missingKeyValues(i)` must be the internal-format value the reader produces for a missing
+ * column: its existence DEFAULT when it has one, else null. The result is cached per set of
+ * missing positions, so those values have to be a function of the positions, which they are: they
+ * come from the scan's schema. `ParquetColumnVector` writes that
+ * default into the output vector, so substituting null instead would filter on a value the scan
+ * never returns and could drop matching rows.
+ *
+ * The predicate has to be evaluated against the substitution rather than skipped, because a null
+ * key does not always mean `false`: a `Coalesce`-wrapped reference still produces a non-null
+ * result, and `XxHash64` is `nullable = false` and hashes a null input to its seed.
+ *
+ * With every key position missing, the result holds no [[BoundReference]] at all and
+ * [[evalAllMissing]] can read off its constant truth value.
+ */
+ def rewriteForMissingKeys(
+ missingKeyLocalPositions: Array[Int],
+ missingKeyValues: Array[Any]): ParquetStorageFilter = {
+ require(missingKeyLocalPositions.length == missingKeyValues.length,
+ "missingKeyLocalPositions and missingKeyValues must have the same length")
+ rewrites.computeIfAbsent(missingKeyLocalPositions.toSeq,
+ _ => rewrite(missingKeyLocalPositions, missingKeyValues))
+ }
+
+ private def rewrite(
+ missingKeyLocalPositions: Array[Int],
+ missingKeyValues: Array[Any]): ParquetStorageFilter = {
+ val substitution = missingKeyLocalPositions.zip(missingKeyValues).toMap
+ val presentPositions = keyColumnIndices.indices.filterNot(substitution.contains)
+ val newPosOf = presentPositions.zipWithIndex.toMap
+ val rewritten = boundExpression.transform {
+ case b: BoundReference if substitution.contains(b.ordinal) =>
+ Literal(substitution(b.ordinal), b.dataType)
+ case b: BoundReference => BoundReference(newPosOf(b.ordinal), b.dataType, b.nullable)
+ }
+ val newKeyColumnIndices = presentPositions.map(keyColumnIndices(_)).toArray
+ new ParquetStorageFilter(newKeyColumnIndices, rewritten, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * When every key column is missing (i.e. [[keyColumnIndices]] is empty after
+ * [[rewriteForMissingKeys]]), the bound expression is fully constant. Evaluates it and returns
+ * `true` iff the predicate is literally true; a null or false result is interpreted as "drop
+ * every row" by the reader.
+ */
+ def evalAllMissing(): Boolean = {
+ require(keyColumnIndices.isEmpty, "evalAllMissing only valid when all key columns are missing")
+ boundExpression.eval(InternalRow.empty) == true
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `extractStorageFilters` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = StorageFilterMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted.
+ //
+ // `sorted` is load-bearing. Both `keyColumnIndices` and the remapped references derive from
+ // this list, so any order would keep those two consistent, but the reader's emit path does
+ // not go through the remapping: it pairs the k-th key slot of the output batch with key-row
+ // position k, which is the identity only while this list is ascending.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val unsupported = originalOrdinals.map(requestedSchema.fields(_))
+ .filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type. This is the single authority on
+ * that: [[isSupportedStorageFilter]] consults it for what the planner asks, and [[create]]
+ * re-checks it, so the reader's per-type value copier
+ * (`VectorizedParquetRecordReader.copierFor`) is only ever asked for a type listed here. Adding
+ * a type here without teaching `copierFor` about it turns a planning-time rejection into a task
+ * failure.
+ *
+ * Narrower than `AtomicType`, for two different reasons:
+ * - `VariantType` cannot be supported: its Parquet representation is a group, not a primitive
+ * leaf, so phase 1 has nothing flat to read it into. (It is unreachable anyway, since
+ * `HashExpression.checkInputDataTypes` rejects variant, so no bloom can be built on one.)
+ * - `GeometryType` and `GeographyType` could be supported. Both map to a primitive Parquet
+ * BINARY and both are handled by `WritableColumnVector.isArray`, so the existing byte-array
+ * copier would work. But no bloom can currently reference them: `HashExpression`'s codegen
+ * type dispatch has no case for either, so hashing one fails at codegen. They are left out
+ * until something can actually produce such a filter.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = dt match {
+ case _: BooleanType | _: ByteType | _: ShortType | _: IntegerType | _: LongType => true
+ case _: FloatType | _: DoubleType | _: DecimalType => true
+ case _: DateType | _: TimestampType | _: TimestampNTZType | _: TimeType => true
+ case _: YearMonthIntervalType | _: DayTimeIntervalType => true
+ // StringType also covers CharType and VarcharType, which extend it.
+ case _: StringType | _: BinaryType => true
+ case _ => false
+ }
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner.
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ // The whole conjunct has to be the bloom, not something with a bloom nested under an OR or a
+ // NOT: the reader evaluates the expression it is given and treats a false as "drop this row".
+ // Every reference is checked, not just the ones on the value side, because [[create]] binds
+ // and type-checks all of them.
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) && canEvaluateOnEveryRow(bloom)
+ case _ => false
+ }
+
+ /**
+ * Whether the reader may evaluate `bloom`'s value side on any row of the files it reads.
+ *
+ * It has to ask, because the reader evaluates the predicate on every row the pushed data filter
+ * left, while in the plan the conjunct ran after the ones ahead of it and was skipped for the
+ * rows they rejected. A `CAST(s AS BIGINT)` key, which `InjectRuntimeFilter` builds for a
+ * string-to-long join, then throws in ANSI mode on a row an earlier conjunct would have dropped,
+ * a query that succeeds without this feature.
+ *
+ * So the value side must be a hash of expressions that cannot fail on any input, which
+ * [[ExprUtils.canEvaluateUnconditionally]] decides. `XxHash64` and the membership test itself are
+ * total for every type they accept at analysis time.
+ *
+ * This is a question about the conjunct as the planner holds it, with attribute references for
+ * leaves. It is not one to ask of a bound expression, which that whitelist does not admit.
+ */
+ private def canEvaluateOnEveryRow(bloom: BloomFilterMightContain): Boolean =
+ bloom.valueExpression match {
+ case hash: XxHash64 => hash.children.forall(ExprUtils.canEvaluateUnconditionally)
Review Comment:
Done the way you suggest, and it replaces the whitelist gate from round 2.
The reader fails open: an evaluation error gives the filter up for that row group, so every row the pushed data filter kept goes to the output and the post-scan `Filter` evaluates the conjuncts in the plan's own order. Either an earlier one drops the row before this expression runs, or it does not and the query fails the way it would have without this feature. The gate now asks only for a deterministic expression with no subquery, so a cast key is pushed, including the `int` joined with `bigint` case.
The catch is narrow: a `RuntimeException` that is a `SparkThrowable` and not an internal error. A defect in the reader still fails the query. And no user code reaches this point, because `InjectRuntimeFilter` will not build a bloom over anything its `isSimpleExpression` calls unsimple, which excludes both UDF kinds, `Invoke`, `JsonToStructs` and the LIKE and REGEXP families.
Tested end to end in ANSI mode with the non-numeric row in the same row group as matching ones, so no statistic prunes it away: ablating the fail-open makes that test fail with `CAST_INVALID_INPUT`, which is how the scenario was confirmed rather than assumed. c29625f
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayOutputStream, File}
+import java.net.URI
+import java.time.LocalTime
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapreduce.Job
+import org.apache.parquet.column.Encoding
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetInputFormat, ParquetOutputFormat}
+
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, XxHash64}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FileSourceScanLike, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests the late-materialization path of [[VectorizedParquetRecordReader]] driven by a
+ * [[ParquetStorageFilter]]. Writes small multi-row-group parquet files, wires a hand-built filter
+ * into the reader, and asserts correctness + the two storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold))
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false), StructField("v", StringType, nullable = false)))
+ ParquetStorageFilter.create(Seq(expr), requested, metrics)
+ }
+
+ // Writes a single-column (just `k`) parquet file for the supplied key type via Spark's
+ // {@code Encoder}.
+ private def writeKeyOnlyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(dir, spark.createDataset(keys).toDF("k"), rowGroupSize)
+
+ // Reads a key-only file, returning the survivor keys and the reader. The {@code extract} function
+ // pulls one value at a time from the batch's key column.
+ private def readKeyOnlyAll[T](
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T,
+ capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k"), storageFilter, (batch, i) => extract(batch.column(0), i),
+ capacity)
+
+ // Builds a `k >= threshold` storage filter bound to position 0 against a key-only schema of the
+ // given key type.
+ private def keyOnlyAtLeastFilter(
+ threshold: Literal,
+ keyType: DataType,
+ metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, keyType, nullable = false), threshold)
+ val requested = StructType(Seq(StructField("k", keyType, nullable = false)))
+ ParquetStorageFilter.create(Seq(expr), requested, metrics)
+ }
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows in one row group: parquet's first row-group size check is at record 100, so
+ // `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = keyAtLeastFilter(1000L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf,
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(rgSkipped.value > 0,
+ s"expected at least one row group skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedRg.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${rowsExcludedRg.value}")
+ assert(rowsExcludedPf.value == 0,
+ s"no partial-row-group filtering expected; got ${rowsExcludedPf.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(bytesAvoidedRg.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got ${bytesAvoidedRg.value}")
+ assert(bytesAvoidedPf.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got ${bytesAvoidedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val filter = keyAtLeastFilter(0L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped, rowsExcludedWithinRowGroup = rowsExcludedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(rgSkipped.value == 0, s"nothing should be skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedPf.value == 0,
+ s"nothing should be filtered; got ${rowsExcludedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("mixed: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf))
+ val (result, reader) = readAll(path, filter)
+ try {
+ // Output is exact: VectorizedColumnReader uses PageReadStore.getRowIndexes (driven by
+ // our finalRanges) to skip rows within partial pages, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it: a row group that was neither skipped whole nor emitted has
+ // to have its rows counted there.
+ assert(rowsExcludedPf.value > 0,
+ s"expected rows excluded inside a kept row group; got ${rowsExcludedPf.value}")
+ assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == rows.size,
+ s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " +
+ s"should account for all ${rows.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key-only projection: phase 2 is skipped and both byte-avoided metrics are zero") {
+ // When the projected schema contains only the bloom key, phase 2 is skipped entirely
+ // (`nonKeyColumns == null` in the reader). All output rows come from the per-key-column
+ // queues populated in phase 1. Total bytes read match the no-storage-filter path (phase 1 reads
+ // the key column once instead of phase 2 re-reading it), so both `avoided` metrics are zero:
+ // there are no non-key bytes to skip. This is the shape the design notes call the biggest win,
+ // so it must not be the shape that pays for metrics.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup")
+ val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup")
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ // k >= 195 -> last 6 keys survive; preceding row groups skipped or page-pruned.
+ val filter = keyOnlyAtLeastFilter(Literal(195L), LongType, StorageFilterMetrics(
+ rowGroupsSkipped = rgSkipped,
+ rowsExcludedByRowGroup = rowsExcludedRg,
+ rowsExcludedWithinRowGroup = rowsExcludedPf,
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i))
+ try {
+ val expected = keys.filter(_ >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact survivor keys; got ${result.sorted}, expected ${expected.toSeq.sorted}")
+ assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}")
+ assert(rowsExcludedRg.value > 0,
+ s"a skipped row group must count its rows too; got ${rowsExcludedRg.value}")
+ // The all-keys path takes its kept-row count from `finalRowCount` rather than from a
+ // phase-2 page store, so this identity is the only thing that checks that arithmetic.
+ assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == keys.size,
+ s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " +
+ s"should account for all ${keys.size} rows")
+ // For an all-keys projection, the no-storage-filter path would have read the same key
+ // column phase 1 reads. There are no non-key bytes to avoid; both metrics are 0.
+ assert(bytesAvoidedRg.value == 0,
+ s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedRg.value}")
+ assert(bytesAvoidedPf.value == 0,
+ s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedPf.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("multi-batch emit: survivor count exceeds capacity") {
+ // Drive the reader at capacity = 16 with a row group of 100 surviving rows. Exercises:
+ // - The per-key-column queue holding multiple full-capacity vectors plus a partial tail.
+ // - pendingCloseKeyVectors getting closed at the start of every subsequent emit.
+ // - The per-emit ColumnarBatch reconstruction running ceil(100/16) = 7 times.
+ withTempDir { dir =>
+ val keys = (1L to 100L)
+ // Big rowGroupSize so all 100 rows fit in one row group.
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 64 * 1024L)
+ // Filter accepts every row so the queue is fully populated.
+ val filter = keyOnlyAtLeastFilter(Literal(0L), LongType)
+ val (result, reader) =
+ readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i), capacity = 16)
+ try {
+ assert(result == keys.toSeq,
+ s"expected all keys returned in order across multiple batches; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("int key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the IntegerType branch of ValueCopier.
+ withTempDir { dir =>
+ val keys = (1 to 100)
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyOnlyAtLeastFilter(Literal(90), IntegerType)
+ val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getInt(i))
+ try {
+ assert(result.toSet == keys.filter(_ >= 90).toSet,
+ s"expected int keys >= 90; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("string key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the StringType branch of ValueCopier (variable-length byte copy via putByteArray).
+ withTempDir { dir =>
+ val keys = (1 to 20).map(i => f"k$i%03d")
+ val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyOnlyAtLeastFilter(
+ Literal.create("k015", StringType), StringType)
+ val (result, reader) =
+ readKeyOnlyAll(path, filter, (vec, i) => vec.getUTF8String(i).toString)
+ try {
+ assert(result.toSet == keys.filter(_ >= "k015").toSet,
+ s"expected string keys >= 'k015'; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction: extractStorageFilters pre-checks each one, and
+ // by the time create runs the conjunct is gone from the post-scan Filter, so a soft rejection
+ // would silently return rows the filter excludes. create fails instead.
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+
+ // Nothing to push: the caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ ParquetStorageFilter.create(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+ }
+
+ // Serializes a [[BloomFilter]] to bytes suitable for a [[Literal]].
+ private def bloomBytes(bf: BloomFilter): Array[Byte] = {
+ val out = new ByteArrayOutputStream()
+ bf.writeTo(out)
+ out.toByteArray
+ }
+
+ // Computes `XxHash64(v)` using the same seed Spark uses for runtime bloom filters.
+ private def xxHash64(v: Any, dt: DataType): Long = {
+ new XxHash64(Seq(Literal(v, dt))).eval(InternalRow.empty).asInstanceOf[Long]
+ }
+
+ test("rewriteForMissingKeys: regular equi-join bloom probes the null key's hash") {
+ // Regular equi-join: the runtime bloom is BloomFilterMightContain(bloom, XxHash64(key)).
+ // XxHash64 is a HashExpression, so it is nullable = false and hashes a null input to its SEED
+ // rather than producing null. Substituting Literal(null) therefore leaves a concrete probe for
+ // `xxHash64(null)`, and whether the file is kept depends on whether that hash is in the bloom.
+ // Both directions are asserted so the outcome does not hinge on a lucky bloom miss.
+ val nullKeyHash = xxHash64(null, LongType)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+
+ def rewriteWithBloomContaining(hashes: Long*): ParquetStorageFilter = {
+ val bf = BloomFilter.create(10, 128)
+ hashes.foreach(bf.putLong)
+ val expr = BloomFilterMightContain(
+ Literal(bloomBytes(bf), BinaryType),
+ new XxHash64(Seq(BoundReference(0, LongType, nullable = true))))
+ val filter = ParquetStorageFilter.create(Seq(expr), requested)
+ filter.rewriteForMissingKeys(Array(0), Array(null))
+ }
+
+ val dropped = rewriteWithBloomContaining(xxHash64(42L, LongType))
+ assert(dropped.keyColumnIndices.isEmpty,
+ "all key positions are missing; keyColumnIndices should be empty")
+ assert(!dropped.evalAllMissing(),
+ "the null key's hash is not in the bloom, so evalAllMissing must return false " +
+ "(skip the file)")
+
+ val kept = rewriteWithBloomContaining(nullKeyHash)
+ assert(kept.evalAllMissing(),
+ "the null key's hash IS in the bloom, so evalAllMissing must return true (keep the file)")
+ }
+
+ test("rewriteForMissingKeys: missing key with an existence DEFAULT probes the default's hash") {
+ // A missing column that has a non-null existence DEFAULT is materialized by
+ // ParquetColumnVector as that default, not as null. The predicate must therefore be evaluated
+ // against the default. Evaluating against null could skip a whole file whose rows all match.
+ val defaultValue = 7L
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(defaultValue, LongType))
+ val expr = BloomFilterMightContain(
+ Literal(bloomBytes(bf), BinaryType),
+ new XxHash64(Seq(BoundReference(0, LongType, nullable = true))))
+ // Substituting the default keeps the file, because the default's hash is in the bloom.
+ assert(ParquetStorageFilter.create(Seq(expr), requested)
+ .rewriteForMissingKeys(Array(0), Array(defaultValue)).evalAllMissing(),
+ "substituting the existence default must probe the default's hash and keep the file")
+ // Substituting null instead would drop it, the bug this guards against. A second filter,
+ // because a rewrite is cached per set of missing positions: one scan always substitutes the
+ // same values for them.
+ assert(!ParquetStorageFilter.create(Seq(expr), requested)
+ .rewriteForMissingKeys(Array(0), Array(null)).evalAllMissing(),
+ "sanity check: substituting null probes a different hash and would drop the file")
+ }
+
+ test("rewriteForMissingKeys: null-safe equi-join bloom keeps the file") {
+ // Null-safe equi-join: ExtractEquiJoinKeys rewrites `a <=> b` so the join key becomes
+ // Coalesce(key, default). After rewriteForMissingKeys substitutes null for the
+ // BoundReference, the coalesce produces `default` (0L here) and the bloom probe checks
+ // whether the default's hash is in the bloom (which it is, matching what the creation side
+ // would have inserted for its own null rows). Expected result: keep the file.
+ val defaultHash = xxHash64(0L, LongType)
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(defaultHash)
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ val expr = BloomFilterMightContain(
+ bloomLit,
+ new XxHash64(Seq(Coalesce(Seq(
+ BoundReference(0, LongType, nullable = true),
+ Literal.default(LongType))))))
+
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(expr), requested)
+
+ val rewritten = filter.rewriteForMissingKeys(Array(0), Array(null))
+ assert(rewritten.keyColumnIndices.isEmpty,
+ "all key positions are missing; keyColumnIndices should be empty")
+ assert(rewritten.evalAllMissing(),
+ "Coalesce(null, default) yields default; bloom hit, so evalAllMissing must return true")
+ }
+
+ test("rewriteForMissingKeys: partial-missing keys narrow keyColumnIndices and renumber") {
+ // Two keys, one missing, one present. Verify the present key's BoundReference is renumbered to
+ // position 0 in the new layout, and the missing one is substituted with Literal(null). We don't
+ // run the predicate here, only the rewrite's structure.
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(1L, LongType))
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ // Two conjuncts: one on ordinal 0 (missing), one on ordinal 1 (present).
+ val exprs = Seq(
+ BloomFilterMightContain(
+ bloomLit, new XxHash64(Seq(BoundReference(0, LongType, nullable = true)))),
+ BloomFilterMightContain(
+ bloomLit, new XxHash64(Seq(BoundReference(1, LongType, nullable = true)))))
+
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(exprs, requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1))
+
+ val rewritten = filter.rewriteForMissingKeys(Array(0), Array(null))
+ assert(rewritten.keyColumnIndices.toSeq == Seq(1),
+ "only the present key column should remain in keyColumnIndices")
+ val ordinals = rewritten.boundExpression.collect {
+ case b: BoundReference => b.ordinal
+ }
+ assert(ordinals == Seq(0),
+ "the remaining BoundReference (for ordinal 1 in requested schema) should be " +
+ s"renumbered to local position 0; got $ordinals")
+ }
+
+ // Returns the FileSourceScanExec for a parquet read, with `storageFilters` attached. Goes through
+ // Spark's normal planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind, the SQL conf check,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def scanWithStorageFilter(
+ path: String,
+ keyName: String,
+ threshold: Long): FileSourceScanExec = {
+ val df = spark.read.parquet(path).select("k", "v")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == keyName).getOrElse(fail(s"No $keyName in scan output"))
+ val expr = GreaterThanOrEqual(keyAttr, Literal(threshold))
+ scan.copy(storageFilters = Seq(expr))
+ }
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect().map(r => (r.getLong(0), r.getString(1)))
+ }
+
+ test("end-to-end via FileSourceScanExec: conf on, filter applied, metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 195L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sortBy(_._1)}; expected $expected")
+
+ val rgSkipped = scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose: phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ assert(scan.simpleString(200).contains("GreaterThan(v,"),
+ s"the data filter must actually be pushed for this test to mean anything: " +
+ scan.simpleString(200))
+ // Storage filter on `k` (key column).
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(
+ storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(100L))))
+
+ val collected = executePlanCollect(withSF).toSet
+ // Every row satisfies v > 'v_000', so the storage filter alone decides the result.
+ val expected = rows.filter(_._1 >= 100L).toSet
+ assert(collected == expected,
+ s"got ${collected.toSeq.sortBy(_._1)}; expected ${expected.toSeq.sortBy(_._1)}")
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ plan.collect {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ plan.collect {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables (bf1: large, bf2: small with selective filter) for join tests.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 = "large" application side: 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 = "small" creation side: 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ body
+ }
+ }
+
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms >= 1,
+ s"expected >= 1 bloom filter on scan.storageFilters; got $storageBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune: the reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSet == Set(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves bloom filter as post-scan FilterExec when conf is off") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters when conf is off; got $storageBlooms")
+ assert(postScanBlooms >= 1,
+ s"expected bloom in post-scan FilterExec when conf is off; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("the feature still engages when ignoreCorruptFiles is on") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSet == Set((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since the
+ // post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty).getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+
+ test("FileSourceStrategy extraction preserves query results") {
+ withBloomFilterTables {
+ val baseConf = Map(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false")
+ val resultConfOff = withSQLConf(
+ (baseConf + (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false")).toSeq: _*) {
+ runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet
+ }
+ val resultConfOn = withSQLConf(
+ (baseConf + (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true")).toSeq: _*) {
+ runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet
+ }
+ assert(resultConfOn == resultConfOff,
+ s"results differ between conf-on and conf-off: on=$resultConfOn off=$resultConfOff")
+ }
+ }
+
+ Seq(Seq("k", "v"), Seq("k")).foreach { projection =>
+ test("a row group over the splice cap is read the plain way, same rows, more bytes " +
+ s"(projection ${projection.mkString(",")})") {
+ // Past the cap a row group is read the plain way: phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Two things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. And
+ // the fallback must actually have happened, which is observable: it reads the key column a
+ // second time, so it transfers strictly more bytes. Without that second assertion the test
+ // would pass even if the cap never reached the reader.
+ //
+ // The batch size is what makes the cap reachable at all: the reader examines the count only
+ // once a batch worth of survivors per key column has been buffered, so 51 survivors have to
+ // cross that line more than once.
+ //
+ // The `k` projection is the interesting one: an all-keys projection normally skips phase 2
+ // entirely, so past the cap it has to read the key column there like any other column.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val readSchema = StructType(projection.map(fileSchema(_)))
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, fileSchema, new StructType(), readSchema, Nil, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, Map.empty)
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = readerFn(file).asInstanceOf[Iterator[Object]].map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the
+ // row ranges: `k >= 350` keeps a contiguous run, so there is one range to hold.
+ val (plainRows, plainBytes) = run("100b")
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ }
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, fewer than the capacity, so the cap is never weighed and the row
+ // group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped")
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = ParquetStorageFilter.create(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true))),
+ StorageFilterMetrics(rowGroupsSkipped = rgSkipped),
+ maxSplicedRowGroupBytes = 100L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity = 16)
+ try {
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(rgSkipped.value >= 1,
+ s"expected a row group with no survivor at all; got ${rgSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group past the cap charges its second key read against the byte metric") {
+ // Giving splicing up means phase 2 reads the key columns a second time, while the baseline
+ // counts them once, in phase 1. That extra read is a cost against the saving, so the same file
+ // and filter must report less avoided than they do while splicing. Without that term the metric
+ // would credit the feature with bytes it did transfer.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+
+ def avoidedBytes(cap: Long): Long = {
+ val pf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = ParquetStorageFilter.create(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L))),
+ fileSchema, StorageFilterMetrics(bytesAvoidedByPageFiltering = pf), cap)
+ val (_, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => b.column(0).getLong(i), capacity = 16)
+ try pf.value finally reader.close()
+ }
+
+ val spliced = avoidedBytes(64L * 1024 * 1024)
+ val plain = avoidedBytes(100L)
+ assert(spliced > 0, s"page filtering has to avoid something here; got $spliced")
+ assert(plain < spliced,
+ s"the second key read must count against the saving; plain=$plain spliced=$spliced")
+ }
+ }
+
+ test("a row group whose survivors scatter into too many ranges is read without the filter") {
+ // `finalRanges` is built one row at a time, so an alternating filter makes one range per
+ // surviving row, and every column reader phase 2 drives materializes that list for the row
+ // group. Past the cap the filter is given up for that row group and every one of its rows is
+ // emitted, which the post-scan Filter then narrows.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def rowsWithCap(cap: Long, capacity: Int = 4096): Int = {
+ val filter =
+ ParquetStorageFilter.create(Seq(everyOtherRow), fileSchema, StorageFilterMetrics(), cap)
+ val (emitted, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.getRow(i).getLong(0), capacity)
+ try emitted.size finally reader.close()
+ }
+
+ assert(rowsWithCap(64L * 1024 * 1024) == 200,
+ "with room for the ranges, only the surviving rows come back")
+ // The budget is weighed in two places, and the capacity decides which one fires. At 4096 the
+ // 200 survivors never fill an accumulator, so only the check after phase 1 can catch them. At
+ // 16 the in-loop check is reached first, and it has to abandon the buffer and give the filter
+ // up together: keeping one without the other would leave phase 2 reading non-key columns
+ // while emit spliced keys from a queue holding only the first survivors.
+ assert(rowsWithCap(1024L) == 400,
+ "past the cap after phase 1, the filter is given up and the row group comes back whole")
+ assert(rowsWithCap(1024L, capacity = 16) == 400,
+ "and the same past the cap inside phase 1's survivor loop")
+ }
+ }
+
+ test("ANSI mode: a bloom over a cast join key stays in the post-scan Filter") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while a reader would evaluate
+ // it on every row of the ranges the pushed filter left. Extraction therefore has to decline, or
+ // this query would start failing when the conf is turned on.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it: a separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"a cast key must not be extracted.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSet == Set("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // `ParquetStorageFilter.test` evaluates the predicate without calling
+ // `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a
+ // `Nondeterministic` expression, so a non-deterministic conjunct has to stay behind. No
+ // producer builds one today, hence the hand-built plan: `InjectRuntimeFilter`'s blooms hash
+ // join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(xxHash64(42L, LongType))
+ val bloomLit = Literal(bloomBytes(bf), BinaryType)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // Control: the same bloom over the key alone is offered to the scan, so the arms differ in
+ // exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. Two gates reject
+ // this one now: the planner's `deterministic` test, and `canEvaluateUnconditionally` inside
+ // the format's answer, whose whitelist is deterministic expressions only.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyOnlyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for: a dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`: `/` is floating-point division in Spark SQL, which would hand every key
+ // expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k"), rowGroupSize,
+ dictionary = dictionary)
+ }
+
+ // The encodings parquet actually used, over every column chunk of the file. Asserted rather than
+ // assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String): Set[Encoding] = {
+ val reader = ParquetFileReader.open(spark.sessionState.newHadoopConf(), new Path(filePath))
+ try {
+ reader.getFooter.getBlocks.asScala
+ .flatMap(_.getColumns.asScala)
+ .flatMap(_.getEncodings.asScala)
+ .toSet
+ } finally {
+ reader.close()
+ }
+ }
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // selects the plain (non-splicing) vectorized path. Every read helper in this suite goes through
+ // here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need: the caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } { reader =>
+ reader.setStorageFilter(storageFilter)
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // Renders one column value as a string using its *internal* representation, so the splicing path
+ // and the plain path can be compared without going through external type conversion.
+ private def renderValue(vec: ColumnVector, i: Int, dt: DataType): String = {
+ if (vec.isNullAt(i)) {
+ "null"
+ } else {
+ dt match {
+ case BooleanType => vec.getBoolean(i).toString
+ case ByteType => vec.getByte(i).toString
+ case ShortType => vec.getShort(i).toString
+ case IntegerType | DateType | _: YearMonthIntervalType => vec.getInt(i).toString
+ case LongType | TimestampType | TimestampNTZType | _: TimeType |
+ _: DayTimeIntervalType => vec.getLong(i).toString
+ case FloatType => vec.getFloat(i).toString
+ case DoubleType => vec.getDouble(i).toString
+ case d: DecimalType => vec.getDecimal(i, d.precision, d.scale).toString
+ case _: StringType => vec.getUTF8String(i).toString
+ case BinaryType => vec.getBinary(i).mkString(",")
+ case other => fail(s"renderValue does not handle $other")
+ }
+ }
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path: whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.column(0), i, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage: one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why: there is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ } {
+ val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded"
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyOnlyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = StructType(Seq(StructField("k", dt, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, dt))
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // Regression test: TimeType is an AtomicType that passes every planning-time gate (it is
+ // batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a TIME
+ // join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyOnlyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = StructType(Seq(StructField("k", dt, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // This is the contract that keeps FileSourceStrategy, ParquetStorageFilter.create and
+ // VectorizedParquetRecordReader.copierFor in lockstep. A type admitted here but missing from
+ // copierFor turns a planning-time rejection into a task failure.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyOnlyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("two key columns: both accumulators stay aligned with each other and with the data column") {
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300 => a in [100, 150]
+ val bound = And(
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)),
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1), "both key columns should be recognized")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, and the emit path depends on
+ // that: it pairs the k-th key slot of the batch with key-row position k, which is the identity
+ // only while the list is ascending. Nothing else in the suite supplies an unsorted one, because
+ // every other fixture happens to mention its keys in column order. Production does not promise
+ // that: the conjuncts arrive in `afterScanFilters` order, which follows the join. Without the
+ // sort this returns `a` and `b` swapped, with no exception to notice.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // Same predicate as the test above, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // Exercises the `i < isKeyTopLevel.length` branch of the emit loop: the partition slot sits
+ // past the end of isKeyTopLevel and must come from persistentBatchColumns, not the key queues.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite: OffHeapColumnVector.close() frees
+ // the native buffer, so a double close or a read after close is a crash rather than stale
+ // data. capacity = 16 over 100 survivors forces 7 emits, each closing the previous emit's
+ // dequeued key vectors.
+ withTempDir { dir =>
+ val path = writeKeyOnlyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(0L))
+ val requested = StructType(Seq(StructField("k", LongType, nullable = true)))
+ val filter = ParquetStorageFilter.create(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (1L to 100L), s"expected all 100 keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test(s"$mode vectors: row-group skipping and page filtering over a mixed projection") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg")
+ val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf")
+ val filter = keyAtLeastFilter(195L, StorageFilterMetrics(
+ bytesAvoidedByRowGroup = bytesAvoidedRg,
+ bytesAvoidedByPageFiltering = bytesAvoidedPf))
+ val reader = new VectorizedParquetRecordReader(useOffHeap, 4096)
+ reader.setStorageFilter(filter)
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ while (i < batch.numRows()) {
+ collected += ((batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+ i += 1
+ }
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 195L),
+ s"expected exact filtering; got ${collected.map(_._1)}")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and the
+ // partially kept one. Both counters must be non-negative, and together positive.
+ assert(bytesAvoidedRg.value >= 0 && bytesAvoidedPf.value >= 0,
+ s"avoided-byte metrics must never go negative; got rg=${bytesAvoidedRg.value} " +
+ s"pf=${bytesAvoidedPf.value}")
+ assert(bytesAvoidedRg.value + bytesAvoidedPf.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // The per-emit ColumnarBatch is replaced on every nextBatch(), so a consumer holding on to an
+ // earlier getCurrentValue() would read the wrong vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new VectorizedParquetRecordReader(false, 8)
+ try {
+ reader.setStorageFilter(filter)
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution: a column missing from the physical file -----
+
+ test("non-key column missing from a file: byte-avoided metrics tolerate a missing offset index") {
+ // ColumnIndexStore returns a null OffsetIndex for a column that is in the (clipped) requested
+ // schema but absent from the row group, which is exactly what schema evolution produces. The
+ // avoided-bytes walk runs over the full requested schema on every row group of every file, so
+ // without a null guard this is an NPE on the first row group of the older file.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // Older file: (k, v). Newer file: (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect()
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSet
+ val expected = ((150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))).toSet
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // The avoided-bytes counters are what walk the offset index of the missing column. Their
+ // being populated and non-negative is the evidence that the walk ran and coped.
+ val bytesRg = withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value >= 0 && bytesPf.value >= 0,
+ s"avoided-byte metrics must never go negative; got rg=${bytesRg.value} " +
+ s"pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = spark.read.parquet(path).select("k", "v").queryExecution.executedPlan
+ .collect { case s: FileSourceScanExec => s }.head
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Set[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Set[(Long, Option[Long])]) = {
+ val df = spark.sql(s"SELECT id, k FROM $table")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(buildFilter(keyAttr)))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = scan.output.indexWhere(_.name == "id")
+ val kPos = scan.output.indexWhere(_.name == "k")
+ val collected = rowPlan.executeCollect()
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSet
+ (withSF, collected)
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test: the default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Set[(Long, Option[Long])] = Set((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected,
+ s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Set((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got ${rejectsNull.toSeq.sorted}")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Set[(Long, Option[Long])] =
+ Set((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got ${acceptsNull.toSeq.sorted}")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L))))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect().map(r => (r.getLong(0), r.getLong(2))).toSet
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1)).toSet
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got " +
+ s"${collected.toSeq.sorted.take(5)}")
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select("k", "s")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output"))
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(195L))))
+ val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF
+ val collected = rowPlan.executeCollect()
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSet
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k")).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sorted}; expected $expected")
+ }
+ }
+ }
+
+ // ----- Planner gates and the lost-filter invariant -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false: it may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ assert(!subclass.supportsStorageFilter(bloom), "a subclass must not claim support")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ // A key the reader could not evaluate on rows the plan would have excluded. In ANSI mode the
+ // cast throws on a string that is not a number, and the reader evaluates the predicate on every
+ // row the pushed filter left, including those an earlier conjunct would have dropped.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+ assert(!format.supportsStorageFilter(onCast), "a cast key can throw on rows the plan excluded")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The whole lost-filter safety argument rests on this gate: if the reader cannot do late
+ // materialization, the planner must NOT move the bloom out of the post-scan Filter.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one: every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, "k", threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSet
+ assert(keys == (1L to 50L).toSet,
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support ignores a non-empty storageFilters") {
+ // The default `FileFormat.buildReaderWithStorageFilters` body may ignore what it is handed,
+ // because the post-scan Filter still holds it. Unreachable in production, since the planner
+ // asks `supportsStorageFilter` first.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(1L)))
+ val e = intercept[Exception] {
+ new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration())
+ }
+ // This stub implements no reader at all, so the error it fails with is the delegation's, which
+ // is what shows the filters were not rejected.
+ assert(e.getMessage.contains("buildReader is not supported"), e.getMessage)
+ }
+
+ Seq(false, true).foreach { aqe =>
+ test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") {
+ // AQE is on by default in production, and it is where the bloom subquery is planned by
+ // PlanAdaptiveSubqueries rather than PlanSubqueries, which is the path
+ // preparedStorageFilters' ScalarSubquery materialization depends on.
+ withBloomFilterTables {
+ val baseConf = Map(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString)
+ def run(pushdown: Boolean): Set[(Long, Long)] = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet
+ }
+ val off = run(false)
+ val on = run(true)
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ }
+ }
+ }
+
+ test("canonicalization keeps a storage-filter scan distinct from a plain one") {
+ // storageFilters is in doCanonicalize and in the case class equality, which is what stops
+ // exchange and subquery reuse from serving one scan's result to the other. Reuse compares
+ // canonicalized plans, so a scan that filters must not match a plain scan of the same file, and
+ // two scans carrying the same filter must still match.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, "k", threshold = 25L)
+ val plain = withSF.copy(storageFilters = Nil)
+ assert(withSF != plain, "case class equality must take storageFilters into account")
+ assert(!withSF.sameResult(plain),
+ s"a filtering scan must not be reusable as a plain one:\n${withSF.canonicalized}\n" +
+ s"${plain.canonicalized}")
+ // A second, independently planned scan of the same file with the same filter still
+ // matches, so reuse is not disabled wholesale. Its key attribute carries a different
+ // exprId, which is what canonicalization normalizes away.
+ val sameSF = scanWithStorageFilter(path, "k", threshold = 25L)
+ assert(withSF.sameResult(sameSF),
+ s"two scans with the same storage filter must stay reusable:\n" +
+ s"${withSF.canonicalized}\n${sameSF.canonicalized}")
+ val otherSF = scanWithStorageFilter(path, "k", threshold = 30L)
+ assert(!withSF.sameResult(otherSF), "a different threshold is a different result")
+ }
+ }
+ }
+
+ test("whole-stage codegen off: the row-at-a-time path still applies the extracted bloom") {
+ // The planner's extraction gate is `fileFormat.supportBatch`, which does not look at
+ // whole-stage codegen, while `FileSourceScanExec.supportsColumnar` does. So with codegen off
+ // the bloom is still extracted, `returningBatch` is false, and the reader serves the spliced
+ // batch one row at a time. That is the only combination where the planner's gate is weaker
+ // than the runtime's.
+ withBloomFilterTables {
+ val baseConf = Map(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "false")
+ def run(pushdown: Boolean): (Set[(Long, Long)], Int, Boolean) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, result) = runBloomFilterJoin()
+ val columnar = plan.collect { case s: FileSourceScanExec => s.supportsColumnar }
+ (result.map(r => (r.getLong(0), r.getLong(1))).toSet,
+ countBloomFiltersInStorageFilters(plan), columnar.forall(_ == false))
+ }
+ val (off, _, _) = run(false)
+ val (on, storageBlooms, noColumnarScan) = run(true)
+ assert(storageBlooms >= 1, s"the bloom must still be extracted with codegen off; " +
+ s"got $storageBlooms")
+ assert(noColumnarScan, "with codegen off no scan should output columnar batches")
+ assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off")
+ assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing")
+ }
+ }
+
+ test("off-heap column vectors through the planner: spliced values survive the free") {
+ // Off-heap is where the vector lifecycle actually bites: the previous batch's key vectors are
+ // freed at the next nextBatch(), so a stale reference reads released native memory rather than
+ // old bytes. This drives it through the planner, where the batch also crosses
+ // ColumnarToRowExec.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.COLUMN_VECTOR_OFFHEAP_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 150L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 150L).toSet
+ assert(collected == expected,
+ s"got ${collected.size} rows; expected ${expected.size}. " +
+ s"first few: ${collected.toSeq.sortBy(_._1).take(3)}")
+ }
+ }
+ }
+
+ // ----- Page-level pushedFilterRanges (a strict subset of the row group) -----
+
+ test("pushed data filter narrows to a page subset: phase 1 stays aligned with the row indexes") {
+ // Every other fixture writes one page per column per row group, so column-index filtering can
+ // only ever drop whole row groups and `pushedFilterRanges` is always the entire block. That
+ // makes the phase 1 alignment hold trivially, that the r-th row readBatch delivers pairs with
+ // rowIndexIter.nextLong(). With a small page size the data filter narrows
+ // to a page subset, so the two sequences only agree if the pairing is actually correct.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // `v` is correlated with `k`, so a range predicate on v prunes pages, not whole row groups.
+ val df = spark.read.parquet(path).select("k", "v").filter("v >= 'v_0300'")
+ val plan = df.queryExecution.executedPlan
+ val scan = plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec in plan: $plan"))
+ assert(scan.simpleString(200).contains("GreaterThanOrEqual(v,"),
+ s"the data filter must be pushed: ${scan.simpleString(200)}")
+ val keyAttr = scan.output.find(_.name == "k").get
+ // Storage filter keeps a band that starts inside the data filter's surviving range.
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(350L))))
+
+ val collected = executePlanCollect(withSF).toSet
+ val expected = rows.filter(r => r._2 >= "v_0300" && r._1 >= 350L).toSet
+ assert(collected == expected,
+ s"got ${collected.size} rows; expected ${expected.size}. " +
+ s"first few: ${collected.toSeq.sortBy(_._1).take(3)}")
+ }
+ }
+ }
+
+ test("page-subset ranges keep the avoided-byte metrics non-negative") {
+ // The strict-subset branch of compressedBytesForRowRanges (offset index + dictionary page) only
+ // runs when pushedFilterRanges is narrower than the block, which needs a multi-page row group.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v >= 'v_0100'")
+ val scan = df.queryExecution.executedPlan
+ .collect { case s: FileSourceScanExec => s }.head
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(390L))))
+ val collected = executePlanCollect(withSF).toSet
+ assert(collected == rows.filter(_._1 >= 390L).toSet, s"got ${collected.size} rows")
+
+ val bytesRg = withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value >= 0 && bytesPf.value >= 0,
Review Comment:
Both fixed.
The assertions are positive now, with a comment saying why the non-negative form proves nothing: `SQLMetric.add` drops a negative, so no arithmetic can make it fail. A positive total can.
The file is written the way you describe, with the `ParquetFileWriter.writeDataPage` overloads that take no row count, and the fixture asserts its own premise: no chunk of it has an offset index. Three row groups, ordered so the first one's partial read is what discovers the missing index, and the two after that still get phase 1. The middle one is emptied by the filter and skipped whole, the last keeps rows and is read whole. So the test pins both halves, that page-level filtering is lost and that whole-row-group skipping is not. c29625f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4106892022
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
Review Comment:
Done: try-with-resources around the phase-1 acquisition and evaluation, so the store is closed on every exit, including skip, give-up and exception.
The ownership point is right and worth writing down, which the code now does: `readFilteredRowGroup` hands out a store the file reader does not track, unlike `readNextRowGroup`, so nothing else would close it. Phase 2's store is closed by this reader too, at the top of the next row group's load and in `close()`. c29625f
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
Review Comment:
This one cannot arise, and there is now a test for the file that would produce it.
Phase 0 sets the full projection before it asks parquet for the ranges, so the block's `ColumnIndexStore` is built over every projected column, key columns included. `ColumnIndexStoreImpl.create` returns `EMPTY` when any of them has no offset index, and with an empty store `ColumnIndexFilter` narrows nothing, so `pushedFilterRanges` covers the whole block. `readFilteredRowGroup(int, RowRanges)` then short-circuits to `internalReadRowGroup` and consults no index at all. The mixed-index file therefore reaches phase 1 with whole-block ranges: one column without an offset index degrades the whole block, which is what makes the phase-1 read safe.
The new test writes a file with no offset index for any column and asserts phase 1 still runs on it. The phase-2 boundary keeps the catch, because that is where a strict subset is asked for, and the retry there no longer trusts the argument above: if `pushedFilterRanges` were ever a strict subset on such a file, it fails with a message that says so rather than throwing from inside parquet. c29625f
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -492,6 +930,630 @@ private void checkEndOfRowGroup() throws IOException {
totalCountLoadedSoFar += pages.getRowCount();
}
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * single {@link #lateMatReader} with its requested schema mutated per phase:
+ * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via
+ * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges},
+ * evaluate the storage filter per row, build {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since
+ * emit then builds every batch from the key queues alone.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO).
+ * Sets {@link #hitEndOfData} when all row groups have been processed.
+ */
+ private void loadNextRowGroupWithLateMaterialization() throws IOException {
+ while (nextBlockIndex < totalBlockCount) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and
+ // trip parquet's own `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself, which is what
+ // `filterGivenUp` says. A file already known to have no offset index starts there.
+ filterGivenUp = fileHasNoOffsetIndex;
+ spliceCurrentRowGroup = !filterGivenUp;
+ splicedBytes = 0L;
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ lateMatReader.setRequestedSchema(requestedColumns);
+ // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(),
+ // so calling it unconditionally would keep applying column-index filtering after a user
+ // turned it off, which is the escape hatch for a file whose column index is wrong. Every
+ // phase below reads within these ranges, so a wrong column index would cost rows the plain
+ // path would have returned. Phase 2 is unaffected: it selects pages through the offset index,
+ // a separate structure this conf says nothing about.
+ RowRanges pushedFilterRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes
+ // a plain read of this projection would transfer for every row the pushed filter kept. The
+ // null checks only skip work for a caller that drives this reader without a scan's metrics;
+ // FileSourceScanLike creates all five whenever storageFilters is non-empty.
+ // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already
+ // given up reports nothing either way, so it does not pay for the baseline at all.
+ StorageFilterMetrics m = storageFilter.metrics();
+ SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup();
+ SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering();
+ boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp;
+ Map<ColumnPath, ColumnChunkMetaData> blockChunks =
+ needBytes ? chunksByPath(lateMatReader, blockIdx) : null;
+ long nonKeyBaselineBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns,
+ pushedFilterRanges, baselineRows)
+ : 0L;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. Skipped for a row group the filter is already given up for, which
+ // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ if (!filterGivenUp) {
+ lateMatReader.setRequestedSchema(keyOnlyColumns);
+ PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges);
+ if (keyPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already
+ // know pushedFilterRanges selects at least one row. Skipping the block here would drop
+ // its surviving rows from the output, so assert rather than `continue`.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges);
+ if (!filterGivenUp
+ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) {
+ // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in
+ // a single one is only caught here, with its survivors buffered. Those are released,
+ // since the ranges they were spliced against are about to be thrown away.
+ giveUpFilter();
+ }
+ if (!filterGivenUp) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter; skip the block entirely,
+ // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns,
+ // and that cost is not part of the baseline, so nothing is subtracted from it here.
+ recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the
+ // projection is all keys and their values were buffered, since emit then builds every batch
+ // from the key queues alone.
+ long keptRows;
+ long phase2Bytes;
+ PageReadStore dataPages = null;
+ if (nonKeyColumns == null && spliceCurrentRowGroup) {
+ keptRows = finalRowCount;
+ phase2Bytes = 0L;
+ } else {
+ lateMatReader.setRequestedSchema(
+ spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself: it resolves every requested column's offset index before reading
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall: a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // Nothing is checked up front, so a file with no page index still reads with the filter
+ // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a
+ // plain read when the ranges cover the block) or rejects one whole.
+ try {
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ LOG.warn("Not applying the storage filter to {}: reading part of a row group needs a "
+ + "Parquet offset index, and this file was written without a page index for at least "
+ + "one projected column", e, MDC.of(LogKeys.PATH, lateMatReader.getFile()));
+ fileHasNoOffsetIndex = true;
+ giveUpFilter();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ lateMatReader.setRequestedSchema(requestedColumns);
+ dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges,
+ // both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ keptRows = dataPages.getRowCount();
+ // Nothing is computed for a row group whose filter was given up: it read what a plain scan
+ // reads, so the answer is a certain zero. `needBytes`, not just `bytesAvoidedPf != null`,
+ // because that is what built `blockChunks`.
+ if (needBytes && bytesAvoidedPf != null && !filterGivenUp) {
+ phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ nonKeyColumns, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks,
+ keyOnlyColumns, finalRanges, finalRowCount);
+ }
+ } else {
+ phase2Bytes = 0L;
+ }
+ }
+ long filteredRows = baselineRows - keptRows;
+ SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup();
+ if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows);
+ if (bytesAvoidedPf != null && !filterGivenUp) {
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the baseline
+ // after giving splicing up contributes nothing rather than subtracting.
+ bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes);
+ }
+
+ if (dataPages != null) {
+ if (rowIndexGenerator != null) {
+ rowIndexGenerator.initFromPageReadStore(dataPages);
+ }
+ for (int i = 0; i < columnVectors.length; i++) {
+ if (spliceCurrentRowGroup && isKeyTopLevel[i]) {
+ // Key columns are sourced from the queues during emit; skip phase-2 reader init.
+ continue;
+ }
+ initColumnReader(dataPages, columnVectors[i]);
+ }
+ }
+ totalCountLoadedSoFar += keptRows;
+ return;
+ }
+ hitEndOfData = true;
+ }
+
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ private static void recordRowGroupSkipped(
+ StorageFilterMetrics m, long excludedRows, long avoidedBytes) {
+ SQLMetric rgSkipped = m.rowGroupsSkipped();
+ if (rgSkipped != null) rgSkipped.add(1L);
+ SQLMetric rowsExcluded = m.rowsExcludedByRowGroup();
+ if (rowsExcluded != null) rowsExcluded.add(excludedRows);
+ SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup();
+ if (bytesAvoided != null) bytesAvoided.add(avoidedBytes);
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ boolean needBytes = m.bytesAvoidedByRowGroup() != null;
+ if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return;
+ List<ColumnDescriptor> projected = requestedSchema.getColumns();
+ List<BlockMetaData> blocks = lateMatReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses: the rows its column index already excluded were never this filter's to
+ // save. `getRowRanges` is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then. It has to be
+ // guarded the same way phase 0 guards it, since it consults the pushed filter but not the
+ // conf that turns column-index filtering off.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = useColumnIndexFilter
+ ? lateMatReader.getRowRanges(blockIdx)
+ : RowRanges.createSingle(blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = needBytes
+ ? compressedBytesForRowRanges(lateMatReader, blockIdx,
+ chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows)
+ : 0L;
+ recordRowGroupSkipped(m, survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The block's column chunks by path, built once per row group and shared by the byte-metric calls
+ * that consume it, since {@link BlockMetaData} offers no lookup of its own.
+ */
+ private static Map<ColumnPath, ColumnChunkMetaData> chunksByPath(
+ ParquetFileReader reader, int blockIndex) {
+ Map<ColumnPath, ColumnChunkMetaData> chunks = new HashMap<>();
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ chunks.put(chunk.getPath(), chunk);
+ }
+ return chunks;
+ }
+
+ /**
+ * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly
+ * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since
+ * both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block: the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters:
+ * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path
+ * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store
+ * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the
+ * storage filter narrowed, which column-index filtering had no hand in, that guarantee is
+ * an ordering one: phase 2's own read of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct: the reader transfers nothing for them.
+ */
+ private static long compressedBytesForRowRanges(
+ ParquetFileReader reader,
+ int blockIndex,
+ Map<ColumnPath, ColumnChunkMetaData> chunks,
+ List<ColumnDescriptor> columns,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (columns == null || columns.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnDescriptor column : columns) {
+ ColumnPath path = ColumnPath.get(column.getPath());
+ ColumnChunkMetaData chunk = chunks.get(path);
+ if (chunk == null) {
+ // Column is in the (clipped) requested schema but not in this file.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group: the
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges) throws IOException {
+ ensureKeyScratchAllocated();
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length];
+ for (int i = 0; i < readers.length; i++) {
+ readers[i] = new VectorizedColumnReader(
+ keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode,
+ datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion);
+ }
+ ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ // Recomputed rather than taken from the caller: a count that disagreed with this iterator would
+ // silently drop surviving rows, and no post-scan Filter is left to catch that.
+ long remaining = pushedFilterRanges.rowCount();
+ boolean accumulate = true;
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ if (storageFilter.test(keyScratchBatch.getRow(r))) {
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (accumulate) {
+ accumulate = appendSurvivorRowToAccumulators(r);
Review Comment:
Done as described. One check, in the per-survivor loop, over `splicedBytes + rowRangeStateBytes(survivorRangeCount)`, and it runs whether or not splicing is still on. Crossing it releases the key accumulators first, since that is the cheaper concession, and gives the filter up when the ranges alone exceed the cap, which returns immediately so the rest of the range list is never built. Nothing partial is left observable: `giveUpFilter` closes every buffered vector, and the caller then reads the row group over `pushedFilterRanges`, which is what a plain read reads.
The conf doc is aligned with it: two pools, both growing with the survivor count, examined after every surviving row, and what is counted is the values and their per-row overhead rather than the backing arrays.
Tests cover both boundaries, a row group that crosses the cap inside the loop and one whose ranges alone do, plus a file whose first row group splices while a later one does not, since the batch is one object and its key slots have to go back to the vectors phase 2 reads into.
The range half also got much cheaper at the root, from @dongjoon-hyun's comment on the same loop: `ParquetReadState` no longer copies the range list per column reader. c29625f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4084294794
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -421,11 +552,313 @@ public boolean nextBatch() throws IOException {
return true;
}
+ /**
+ * Splicing emit path. Closes the previous emit's dequeued key vectors (releasing survivor memory
+ * incrementally), advances to the next row group if needed via {@link #checkEndOfRowGroup()},
+ * dequeues one survivor key vector per key column, drives non-key column readers for {@code num}
+ * rows, and assembles a fresh {@link ColumnarBatch} interleaving key (dequeued) and non-key
+ * (persistent value-vector) slots in the original projection order.
+ * The per-emit batch is a transient view over vectors owned elsewhere; see {@link #close()} and
+ * {@link #closeSplicingState()}.
+ */
+ private boolean nextBatchSplicing() throws IOException {
Review Comment:
Fair question, and it is now answered in the description under **Design decisions** as well as in the reader's own note.
Re-reading is where this started, and it was measured slower: on TPCDS q37, 3949 ms splicing against 4511 ms re-reading, with the feature off at 3199 ms. The 562 ms between the two is one more read of `cs_item_sk`, and nothing absorbs it on object storage -- parquet caches footers rather than pages, and S3A's default input stream caches nothing, so it is a new GET (the caching stream types are opt-in).
Two corrections to the cost model. The re-read does not remove the `LIMIT` stall, since phase 1 still walks the whole row group to build `finalRanges`. And it would not remove the buffering either, for the same reason -- what it removes is the splicing machinery, which is real but is the part with tests around it.
Your comment did change the planner, though. q37 is an all-keys projection, and on that shape neither design can win: the reader has to read the key column to evaluate the filter on it, so the scan reads the same columns for the same rows as a plain one. That is what 3199 against 3949 shows. Extraction is now dropped when every projected data column is a key column of the filter.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1885,6 +1885,21 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED =
+ buildConf("spark.sql.parquet.storageFilterPushdown.enabled")
+ .doc("If true, allows the vectorized Parquet reader to evaluate runtime storage filters " +
+ "(e.g. bloom filters from join runtime filtering) at the scan level using late " +
+ "materialization: read key columns first, evaluate the filter per row, then read data " +
+ "columns restricted to surviving rows. This is a planning-time decision only: when " +
+ "false, no storage filter is attached to a scan in the first place and the filter is " +
+ "applied as an ordinary post-scan filter instead. Note that the surviving key values of " +
+ "a whole row group are buffered before the first batch of that row group is produced, " +
+ "so a task holds up to one extra copy of the key columns for one row group.")
+ .version("5.0.0")
Review Comment:
Master-only on purpose, so `5.0.0` is right. The reader needs the parquet 1.18 APIs -- `RowRanges` public in `filter2.columnindex`, `RowRanges.builder()`, `addSelectedRow(long)` and a public `ParquetFileReader.getRowRanges(int)` -- and that upgrade (#58120) went to master alone, with `branch-4.x` still on 1.17.0.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -151,6 +152,63 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
}
}
+ /**
+ * Splits `afterScanFilters` into bloom-filter conjuncts that can be pushed to the storage layer
+ * for late materialization (returned as the first element) and the remaining filters that stay as
+ * a post-scan FilterExec (returned as the second element).
+ *
+ * Eligibility (statically checked here so the runtime never silently loses the filter):
+ * - The storage-filter pushdown SQL conf is on.
+ * - The file format is exactly [[ParquetFileFormat]]. Subclasses are excluded on purpose: they
+ * may customize reading by overriding `buildReaderWithPartitionValues`, and attaching storage
+ * filters would route the scan through `ParquetFileFormat`'s own reader instead, silently
+ * dropping whatever the subclass does.
+ * - The vectorized reader is feasible for the schema the reader will actually see, i.e.
+ * `partitionSchema ++ outputDataSchema` -- the same schema `ParquetFileFormat.buildReader`
+ * derives `enableVectorizedReader` from.
+ * - The conjunct is a top-level [[BloomFilterMightContain]] (not nested under OR/NOT).
+ * - The conjunct is deterministic. `ParquetStorageFilter.test` evaluates the predicate without
+ * calling `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a
+ * `Nondeterministic` expression, so a non-deterministic conjunct would fail at task time. No
+ * such bloom exists today, because the only producer is `InjectRuntimeFilter` and a join key
+ * is deterministic, but this gate should not depend on a distant rule.
+ * - The bloom's value-side references are projected data columns whose type the reader's value
+ * copier supports (see [[ParquetStorageFilter.isSupportedKeyType]]).
+ *
+ * If any condition fails, ALL bloom filters stay in the second element to preserve the existing
+ * fallback behavior.
+ */
+ private def extractStorageFilters(
+ afterScanFilters: ExpressionSet,
+ fsRelation: HadoopFsRelation,
+ readDataColumns: Seq[Attribute],
+ outputDataSchema: StructType): (Seq[Expression], ExpressionSet) = {
+ val sparkSession = fsRelation.sparkSession
+ val sqlConf = sparkSession.sessionState.conf
+ if (!sqlConf.parquetStorageFilterPushdownEnabled) return (Nil, afterScanFilters)
+ if (fsRelation.fileFormat.getClass != classOf[ParquetFileFormat]) {
Review Comment:
Done, as `FileFormat.supportsStorageFilter(expr)` defaulting to false. `FileSourceStrategy` no longer imports the parquet package, and the key-type list lives next to the copier that defines it. The subclass exclusion moved into `ParquetFileFormat`'s override, where it reads as a fact about that reader, exactly as you put it.
One deviation: the signature takes only the expression, not the read schema. Everything the format needs is reachable from the expression's references, and schema-level feasibility is `supportBatch`, which the planner asks a few lines above. It can grow the parameter when a second format needs one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4159560267
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/StorageFilterPushdownBenchmark.scala:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import java.io.File
+
+import org.apache.spark.SparkConf
+import org.apache.spark.benchmark.Benchmark
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.BloomFilterMightContain
+import org.apache.spark.sql.execution.{FileSourceScanExec, SparkPlan}
+import org.apache.spark.sql.execution.datasources.parquet.StorageFilterMetrics
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Benchmark to measure storage-filter pushdown in the vectorized Parquet reader. Every case runs
+ * the same join, whose runtime bloom filter is the storage filter, with
+ * `spark.sql.parquet.storageFilterPushdown.enabled` off and on.
+ *
+ * The fact table is written in its join key's order, so which keys a dimension table selects
+ * decides where the surviving rows fall:
+ * - clustered keys put the survivors in a few pages, so the reader skips the other row groups
+ * whole and the other pages of the row group that holds them. These cases show what the
+ * feature saves. The bloom filter's false positives land anywhere, though, so the more keys
+ * it holds, the more pages hold a survivor;
+ * - scattered keys put survivors in every page, so the reader can skip nothing and decodes the
+ * non-key columns range by range. These cases show what the feature costs.
+ * After each case it prints what the storage filter excluded, which does not depend on the
+ * machine.
+ *
+ * To run this benchmark:
+ * {{{
+ * 1. without sbt: bin/spark-submit --class <this class>
+ * --jars <spark core test jar>,<spark catalyst test jar> <spark sql test jar>
+ * 2. build/sbt "sql/Test/runMain <this class>"
+ * 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/Test/runMain <this class>"
+ * Results will be written to "benchmarks/StorageFilterPushdownBenchmark-results.txt".
+ * }}}
+ */
+object StorageFilterPushdownBenchmark extends SqlBasedBenchmark {
+
+ override def getSparkSession: SparkSession = {
+ val conf = new SparkConf()
+ .setAppName(this.getClass.getSimpleName)
+ // Since `spark.master` always exists, overrides this value
+ .set("spark.master", "local[1]")
+ .setIfMissing("spark.driver.memory", "3g")
+ .setIfMissing("spark.executor.memory", "3g")
+ .set(SQLConf.SHUFFLE_PARTITIONS.key, "1")
+ .set("spark.ui.enabled", "false")
+
+ SparkSession.builder().config(conf).getOrCreate()
+ }
+
+ private val numRows = 4L * 1024 * 1024
+ private val payloadColumns = 8
+
+ // The clustered cases select a contiguous range of keys from the middle of the table.
+ private val clustered = Seq("0.1%" -> 0.001, "1%" -> 0.01, "10%" -> 0.1)
+ // The scattered cases select keys by their hash, so the survivors are spread over every page.
+ private val scattered = Seq("3%" -> 3, "25%" -> 25, "50%" -> 50)
+
+ private def prepareTables(dir: File): Unit = {
+ val factPath = new File(dir, "fact").getCanonicalPath
+ // The payload is hashed, so it neither compresses nor dictionary-encodes away, and the bytes a
+ // skipped page saves are real.
+ val payload = (0 until payloadColumns).map(i => s"xxhash64(id, $i) AS v$i")
+ spark.range(0, numRows, 1, 1).selectExpr("id AS k" +: payload: _*).write.parquet(factPath)
+ spark.read.parquet(factPath).createOrReplaceTempView("fact")
+
+ def writeDimension(name: String, predicate: String): Unit = {
+ val path = new File(dir, name).getCanonicalPath
+ // `sel_id` exists for the query's filter on the dimension side, which is what makes the
+ // optimizer build a bloom filter from it at all.
+ spark.range(0, numRows, 1, 1).where(predicate).selectExpr("id AS k", "id AS sel_id")
+ .write.parquet(path)
+ spark.read.parquet(path).createOrReplaceTempView(name)
+ }
+ clustered.foreach { case (label, fraction) =>
+ val start = numRows / 2
+ val end = start + (numRows * fraction).toLong
+ writeDimension(dimension("clustered", label), s"id >= $start AND id < $end")
+ }
+ scattered.foreach { case (label, percent) =>
+ writeDimension(dimension("scattered", label), s"pmod(xxhash64(id), 100) < $percent")
+ }
+ }
+
+ private def dimension(layout: String, label: String): String =
+ s"${layout}_${label.replace("%", "").replace(".", "_")}"
+
+ private def query(dimension: String): String =
+ s"SELECT f.* FROM fact f JOIN $dimension d ON f.k = d.k WHERE d.sel_id >= 0"
Review Comment:
Minor: thank you for the benchmark (R4-10). Each timed iteration runs this whole query on one core: the bloom subquery, both scans, a shuffle and a sort of both sides, and the sort-merge join. The join dominates, so the scattered cases' 0.9X understates what the reader costs, while the scaladoc at L41-42 says these cases "show what the feature costs". The off arm reads the same 4M rows in every case, yet takes 718, 1926 and 3461 ms (JDK 17) at 3%, 25% and 50%, because the join grows with the rows the post-scan Filter keeps. Bounding the off arm's scan and post-scan Filter by the whole 3% off run, the on-minus-off differences of +299 and +349 ms at 25% and 50% put the scan itself at 1.42x and 1.49x or more on JDK 17, and at 1.44x to 1.47x and 1.61x to 1.67x on JDK 21 and 25. The break-even point itself is right, since both arms pay the same join. Could the scattered cases get a join-free arm, e.g. a scan of the fact table under a `BloomFilterMightContain` built once outside the tim
ed body, the way `ParquetStorageFilterSuite.scala:1675-1676` builds one? Or, at least, could the scaladoc and the description give the scan's own ratio?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while evaluating a storage filter, if it must not fall back
+ * on it by giving the filter up. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped, see [[throwIfKilled]]. The error may be the kill's
+ * interrupt surfacing through a UDF that wrapped it. `PythonRunner` reads a kill off the task
+ * context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had the post-scan Filter raised `e`, which it raises as it is, goes out
+ * as itself, so the executor finds it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in a checked wrapper. `FileScanRDD` would take a bare
+ * one for a corrupt file under `ignoreCorruptFiles`, where a plain plan raises it from the
+ * post-scan Filter, outside that catch. The executor finds it under the wrapper and the
+ * FAILED_READ_FILE around that at a `spark.executor.killOnFatalError.depth` of 3 or more, the
+ * default being 5.
+ *
+ * Anything else is the filter's own error, which the post-scan Filter raises again for a row it
+ * does evaluate, so the reader falls back on it.
+ */
+ def rethrowIfMustPropagate(e: Throwable): Unit = rethrowIfMustPropagate(e, fromRead = false)
+
+ /**
+ * As [[rethrowIfMustPropagate]], for an error decoding the key pages, with three differences:
+ * - a plain read raises that from the reader, where `FileScanRDD` wraps it in FAILED_READ_FILE
+ * before the executor looks, so a fatal error deeper in the chain has to sit one level
+ * shallower to count.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ * The reader falls back on those, and the plain read of the same pages meets the corrupt data
+ * again, or allocates what a plain read allocates.
+ */
+ def rethrowIfMustPropagateFromRead(e: Throwable): Unit =
+ rethrowIfMustPropagate(e, fromRead = true)
+
+ private def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new SparkException("A storage filter met a fatal error it must not absorb", internal)
Review Comment:
**With a kill pending, an `InternalError` from the filter leaves the executor running, so "kill or no kill" (L246) does not hold for it.** `out` wraps it in a `SparkException`, which is `NonFatal`, and `FileDataSourceV2.attachFilePath` wraps that in `FAILED_READ_FILE`. The executor matches `case _: InterruptedException | NonFatal(_)` guarded by `task.reasonIfKilled.isDefined` (`Executor.scala:1068-1069`) before its fatal check (`Executor.scala:1144`), so when the job is cancelled, or a speculative sibling finishes, it reports the task killed and the JVM goes on. In a plain plan the post-scan Filter raises the bare `InternalError`, which that case does not match, and the executor exits. The kill-pending block of the new test checks only an OOM (`ParquetStorageFilterSuite.scala:659-672`). It takes a JVM-level `InternalError` and a kill at once, so it is rare. Could `out` wrap it in a fatal error that is not an `InternalError`, e.g. `new UnknownError(msg).initCause(internal)`? Neithe
r `shouldIgnoreCorruptFileException` nor the kill case takes that, the executor finds it at the top, and the depth caveat at L254-258 goes away. Otherwise, could L246 and L254-258 name this exception?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/StorageFilterPushdownBenchmark.scala:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import java.io.File
+
+import org.apache.spark.SparkConf
+import org.apache.spark.benchmark.Benchmark
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.BloomFilterMightContain
+import org.apache.spark.sql.execution.{FileSourceScanExec, SparkPlan}
+import org.apache.spark.sql.execution.datasources.parquet.StorageFilterMetrics
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Benchmark to measure storage-filter pushdown in the vectorized Parquet reader. Every case runs
+ * the same join, whose runtime bloom filter is the storage filter, with
+ * `spark.sql.parquet.storageFilterPushdown.enabled` off and on.
+ *
+ * The fact table is written in its join key's order, so which keys a dimension table selects
+ * decides where the surviving rows fall:
+ * - clustered keys put the survivors in a few pages, so the reader skips the other row groups
+ * whole and the other pages of the row group that holds them. These cases show what the
+ * feature saves. The bloom filter's false positives land anywhere, though, so the more keys
+ * it holds, the more pages hold a survivor;
+ * - scattered keys put survivors in every page, so the reader can skip nothing and decodes the
+ * non-key columns range by range. These cases show what the feature costs.
+ * After each case it prints what the storage filter excluded, which does not depend on the
+ * machine.
+ *
+ * To run this benchmark:
+ * {{{
+ * 1. without sbt: bin/spark-submit --class <this class>
+ * --jars <spark core test jar>,<spark catalyst test jar> <spark sql test jar>
+ * 2. build/sbt "sql/Test/runMain <this class>"
+ * 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/Test/runMain <this class>"
+ * Results will be written to "benchmarks/StorageFilterPushdownBenchmark-results.txt".
+ * }}}
+ */
+object StorageFilterPushdownBenchmark extends SqlBasedBenchmark {
+
+ override def getSparkSession: SparkSession = {
+ val conf = new SparkConf()
+ .setAppName(this.getClass.getSimpleName)
+ // Since `spark.master` always exists, overrides this value
+ .set("spark.master", "local[1]")
+ .setIfMissing("spark.driver.memory", "3g")
+ .setIfMissing("spark.executor.memory", "3g")
+ .set(SQLConf.SHUFFLE_PARTITIONS.key, "1")
+ .set("spark.ui.enabled", "false")
+
+ SparkSession.builder().config(conf).getOrCreate()
+ }
+
+ private val numRows = 4L * 1024 * 1024
+ private val payloadColumns = 8
+
+ // The clustered cases select a contiguous range of keys from the middle of the table.
+ private val clustered = Seq("0.1%" -> 0.001, "1%" -> 0.01, "10%" -> 0.1)
+ // The scattered cases select keys by their hash, so the survivors are spread over every page.
+ private val scattered = Seq("3%" -> 3, "25%" -> 25, "50%" -> 50)
+
+ private def prepareTables(dir: File): Unit = {
+ val factPath = new File(dir, "fact").getCanonicalPath
+ // The payload is hashed, so it neither compresses nor dictionary-encodes away, and the bytes a
+ // skipped page saves are real.
+ val payload = (0 until payloadColumns).map(i => s"xxhash64(id, $i) AS v$i")
+ spark.range(0, numRows, 1, 1).selectExpr("id AS k" +: payload: _*).write.parquet(factPath)
+ spark.read.parquet(factPath).createOrReplaceTempView("fact")
+
+ def writeDimension(name: String, predicate: String): Unit = {
+ val path = new File(dir, name).getCanonicalPath
+ // `sel_id` exists for the query's filter on the dimension side, which is what makes the
+ // optimizer build a bloom filter from it at all.
+ spark.range(0, numRows, 1, 1).where(predicate).selectExpr("id AS k", "id AS sel_id")
+ .write.parquet(path)
+ spark.read.parquet(path).createOrReplaceTempView(name)
+ }
+ clustered.foreach { case (label, fraction) =>
+ val start = numRows / 2
+ val end = start + (numRows * fraction).toLong
+ writeDimension(dimension("clustered", label), s"id >= $start AND id < $end")
+ }
+ scattered.foreach { case (label, percent) =>
+ writeDimension(dimension("scattered", label), s"pmod(xxhash64(id), 100) < $percent")
+ }
+ }
+
+ private def dimension(layout: String, label: String): String =
+ s"${layout}_${label.replace("%", "").replace(".", "_")}"
+
+ private def query(dimension: String): String =
+ s"SELECT f.* FROM fact f JOIN $dimension d ON f.k = d.k WHERE d.sel_id >= 0"
+
+ private def factScan(plan: SparkPlan): FileSourceScanExec =
+ plan.collectFirst {
+ case scan: FileSourceScanExec if scan.output.exists(_.name == "v0") => scan
+ }.getOrElse(throw new IllegalStateException(s"No fact table scan in $plan"))
+
+ private def withPushdown[T](enabled: Boolean)(f: => T): T =
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> enabled.toString)(f)
+
+ // Runs the query once, untimed, and returns the fact scan with its metrics filled in.
+ private def executedFactScan(sql: String, enabled: Boolean): FileSourceScanExec =
+ withPushdown(enabled) {
+ val df = spark.sql(sql)
+ val scan = factScan(df.queryExecution.executedPlan)
+ val hasBloom = df.queryExecution.executedPlan.exists { node =>
+ node.expressions.exists(_.exists(_.isInstanceOf[BloomFilterMightContain]))
+ }
+ assert(hasBloom, s"The join must have a runtime bloom filter:\n${df.queryExecution}")
+ assert(scan.storageFilters.nonEmpty == enabled,
+ s"The fact scan must have a storage filter exactly when pushdown is on:\n$scan")
+ df.queryExecution.toRdd.foreach(_ => ())
+ scan
+ }
+
+ private def runCase(title: String, dimension: String): Unit = {
+ val sql = query(dimension)
+ val benchmark = new Benchmark(title, numRows, minNumIters = 3, output = output)
+ Seq(false, true).foreach { enabled =>
+ benchmark.addCase(s"Storage filter pushdown ${if (enabled) "on" else "off"}") { _ =>
+ withPushdown(enabled)(spark.sql(sql).noop())
+ }
+ }
+ benchmark.run()
+
+ val off = executedFactScan(sql, enabled = false)
Review Comment:
Nit: `off` is used only for its `numOutputRows`, which is `numRows` by construction: the fact side's only pushed filter is the inferred `IsNotNull(k)`, which prunes nothing, and the bloom stays above the scan. All six cases in the three results files print 4194304. So each case runs the whole join once more, untimed, for a constant, about 8.2 to 8.6 s per JDK run. Could the off arm keep its two plan assertions, which need only `executedPlan`, skip `toRdd.foreach`, and print `numRows`?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while evaluating a storage filter, if it must not fall back
+ * on it by giving the filter up. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped, see [[throwIfKilled]]. The error may be the kill's
+ * interrupt surfacing through a UDF that wrapped it. `PythonRunner` reads a kill off the task
+ * context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had the post-scan Filter raised `e`, which it raises as it is, goes out
+ * as itself, so the executor finds it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in a checked wrapper. `FileScanRDD` would take a bare
+ * one for a corrupt file under `ignoreCorruptFiles`, where a plain plan raises it from the
+ * post-scan Filter, outside that catch. The executor finds it under the wrapper and the
+ * FAILED_READ_FILE around that at a `spark.executor.killOnFatalError.depth` of 3 or more, the
+ * default being 5.
+ *
+ * Anything else is the filter's own error, which the post-scan Filter raises again for a row it
+ * does evaluate, so the reader falls back on it.
+ */
+ def rethrowIfMustPropagate(e: Throwable): Unit = rethrowIfMustPropagate(e, fromRead = false)
+
+ /**
+ * As [[rethrowIfMustPropagate]], for an error decoding the key pages, with three differences:
+ * - a plain read raises that from the reader, where `FileScanRDD` wraps it in FAILED_READ_FILE
+ * before the executor looks, so a fatal error deeper in the chain has to sit one level
+ * shallower to count.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ * The reader falls back on those, and the plain read of the same pages meets the corrupt data
+ * again, or allocates what a plain read allocates.
+ */
+ def rethrowIfMustPropagateFromRead(e: Throwable): Unit =
+ rethrowIfMustPropagate(e, fromRead = true)
+
+ private def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new SparkException("A storage filter met a fatal error it must not absorb", internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ if (Executor.isFatalError(e, if (fromRead) fatalErrorDepth - 1 else fatalErrorDepth)) {
+ // The one `Executor.isFatalError` found, which no SparkOutOfMemoryError can precede.
+ val fatal = Iterator.iterate(e)(_.getCause).find(Utils.isFatalError).get
+ if (!corrupt(fatal)) throw out(fatal)
+ }
+ }
+
+ /**
+ * Whether `e` is how `WritableColumnVector.reserve` reports a vector that cannot grow: a plain
+ * `RuntimeException`, caused by an `OutOfMemoryError` or by nothing, for a capacity past what a
+ * vector can hold.
+ */
+ private def isVectorGrowthFailure(e: Throwable): Boolean =
+ e.getClass == classOf[RuntimeException] &&
Review Comment:
Minor: this matches the exact class `WritableColumnVector.reserve` throws today, `new RuntimeException(message, cause)` (`WritableColumnVector.java:159`), and its only test builds that exception by hand (`ParquetStorageFilterSuite.scala:653`). If `throwUnsupportedException` moves to an error-class exception such as `SparkRuntimeException`, as many of Spark's errors have, `ColumnarBatchSuite`'s `intercept[RuntimeException]` still passes, this check stops matching, and L291 finds the OOM one level down and rethrows it. A phase-1 key vector that cannot grow would then stop the executor instead of falling back, with every test green. The `e.getCause == null` arm makes no difference, since such an exception has no fatal cause and falls back at L291 anyway. Could the test take the exception from a real `reserve` call, e.g. `reserve(-1)`, which goes through the same `throwUnsupportedException`, and assert its class? And could the null arm go?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1051 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
Review Comment:
Nit: this holds with `reserve`'s default doubling only. With `spark.sql.inMemoryColumnarStorage.hugeVectorThreshold` set, -1 by default, and `spark.sql.inMemoryColumnarStorage.hugeVectorReserveRatio` above 2, both public and unchecked, a child whose request reaches the threshold grows by that ratio (`WritableColumnVector.java:131-134`). It then allocates 2 * ratio bytes per value byte while this charges 4, and the up-front `reserve(sizeTo[i])` at L906 allocates as much while L907 prepays 4 * `sizeTo[i]`. The same goes for "doubles what it is asked for" at L876. The conf doc already calls the count an estimate, so only these comments overstate it. Also, L470 spells `DEFAULT_CHILD_BYTES_PER_ROW` as the literals `4 + 4`. Could these comments say "with `reserve`'s default doubling", and could L470 read `1 + 4 + 4 + 2 * DEFAULT_CHILD_BYTES_PER_ROW`?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1051 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages {@link #reportOnce} has already reported. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /** Whether the reader has reported an offset index parquet could not read. */
+ private boolean reportedUnreadableOffsetIndex;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (!reportedUnreadableOffsetIndex) {
+ reportedUnreadableOffsetIndex = true;
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn("Reading a row group of {} without page-level storage filtering, because "
+ + "parquet could not read the offset index of a column the read needs ({}). The "
+ + "other row groups are filtered as usual. Reported once per split",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(t);
+ reportOnce(t, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
Review Comment:
**After a vector-growth failure, this fallback keeps the grown scratch vector, so phase 2 does not allocate only what a plain read allocates.** The doc at `ParquetStorageFilter.scala:272-276` lets a key vector that cannot grow fall back, since "phase 1 decodes into vectors only this reader allocates", and says the plain read of the same pages then "allocates what a plain read allocates". But nothing here releases `keyScratchVectors`: `abandonSplicing()` frees only the survivor buffers, `reset()` frees nothing at the default `hugeVectorThreshold` of -1 (`WritableColumnVector.java:94`), and the scratch vectors are closed only in `close()` (L276). A failed `reserve` also keeps the old buffer, since the new one is allocated before it replaces the old.
Phase 2 then reads the key column into the persistent vector, in the same batches with the same values, and grows it through the same sizes while the scratch child still holds its last capacity. So for a string or binary key with large values, phase 2 can fail where a plain read, one such buffer lighter, fits, and `FAILED_READ_FILE(RuntimeException(OOM))` then stops the executor at the default `spark.executor.killOnFatalError.depth`, or skips the rest of the file under `ignoreCorruptFiles`. Could this catch release the scratch vectors before it returns, e.g. `closeAll(keyScratchVectors); allocateKeyScratch();`? `allocateKeyScratch()` rebuilds `keyScratchBatch` from the persistent vectors, so it is safe to call again, and doing it on every decoding fallback costs one capacity-sized allocation.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1051 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages {@link #reportOnce} has already reported. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /** Whether the reader has reported an offset index parquet could not read. */
+ private boolean reportedUnreadableOffsetIndex;
Review Comment:
Nit: this round added two once-per-split gates to this class, `reportedMessages` behind `reportOnce` (L171) and this boolean for the one WARN that cannot go through `reportOnce`, since it logs no throwable and adds a `REASON`. Its text also sits apart from the other four messages (L962-982). Could it be a constant next to them, logged under `if (reportedMessages.add(...))`, so that this boolean can go? Also, nothing pins "once": the damaged-index test damages a single block, so the suite stays green without any gate.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1051 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages {@link #reportOnce} has already reported. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /** Whether the reader has reported an offset index parquet could not read. */
+ private boolean reportedUnreadableOffsetIndex;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (!reportedUnreadableOffsetIndex) {
+ reportedUnreadableOffsetIndex = true;
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn("Reading a row group of {} without page-level storage filtering, because "
+ + "parquet could not read the offset index of a column the read needs ({}). The "
+ + "other row groups are filtered as usual. Reported once per split",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(t);
Review Comment:
**This catch no longer goes through `giveUpOnError`, and nothing fails if this line goes.** R7-12 asked for one helper so that a new catch cannot take the report without the rethrow, and in round 8 all three catches called `giveUpOnError`, whose javadoc said "Every catch that gives the filter up goes through here". This round calls `rethrowIfMustPropagateFromRead` and `reportOnce` here by hand, and the javadoc at L984-989 now covers only the two evaluation sites. Deleting this line, or calling `rethrowIfMustPropagate` instead, leaves the suite green: every reader test that raises a kill or a fatal error does so through `OnKey`, which is the evaluation path, the depth test calls the helpers directly, and the corrupt-page test (`ParquetStorageFilterSuite.scala:1308`) raises a non-fatal error that falls back either way. Since this catch now takes `Throwable`, this line is all that lets an OOM or a kill out of key-page decoding. Could `giveUpOnError` take the variant, e.g. a `fromRead
` flag, so that all three catches go through it again? And could a test pin this site, e.g. the corrupt key page read under a spied `TaskContext` whose `isInterrupted()` turns true only at this catch, after the per-chunk check at L745 has passed?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1051 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages {@link #reportOnce} has already reported. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /** Whether the reader has reported an offset index parquet could not read. */
+ private boolean reportedUnreadableOffsetIndex;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
Review Comment:
Minor: R8-6 noted that with a pushed data filter, `fileReader.getColumnIndexStore(i).getOffsetIndex(path)` answers per block with no IO. The per-block part is in, but only through the catch at L567. A block whose offset index parquet could not read holds the `EMPTY` store, which `getFilteredRecordCount()` cached at initialize and whose `getOffsetIndex` throws for any path. So such a block is known here, yet phase 1 still decodes and evaluates it past its first survivor and buffers survivors up to the cap, and phase 2 then fails its narrowed read and reads the block whole. Could `canNarrowRowGroup` also probe the cached store when a filter is pushed (`FilterCompat.isFilteringRequired(readOptions.getRecordFilter())`), keeping the catch as the backstop when none is? `ParquetBlockAccounting` already reads `ColumnIndexStore`, so the probe adds no new internal type, and the damaged-index test would need a variant without a pushed filter to keep reaching the catch.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while evaluating a storage filter, if it must not fall back
+ * on it by giving the filter up. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped, see [[throwIfKilled]]. The error may be the kill's
+ * interrupt surfacing through a UDF that wrapped it. `PythonRunner` reads a kill off the task
+ * context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had the post-scan Filter raised `e`, which it raises as it is, goes out
+ * as itself, so the executor finds it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in a checked wrapper. `FileScanRDD` would take a bare
+ * one for a corrupt file under `ignoreCorruptFiles`, where a plain plan raises it from the
+ * post-scan Filter, outside that catch. The executor finds it under the wrapper and the
+ * FAILED_READ_FILE around that at a `spark.executor.killOnFatalError.depth` of 3 or more, the
+ * default being 5.
+ *
+ * Anything else is the filter's own error, which the post-scan Filter raises again for a row it
+ * does evaluate, so the reader falls back on it.
+ */
+ def rethrowIfMustPropagate(e: Throwable): Unit = rethrowIfMustPropagate(e, fromRead = false)
+
+ /**
+ * As [[rethrowIfMustPropagate]], for an error decoding the key pages, with three differences:
+ * - a plain read raises that from the reader, where `FileScanRDD` wraps it in FAILED_READ_FILE
+ * before the executor looks, so a fatal error deeper in the chain has to sit one level
+ * shallower to count.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ * The reader falls back on those, and the plain read of the same pages meets the corrupt data
+ * again, or allocates what a plain read allocates.
+ */
+ def rethrowIfMustPropagateFromRead(e: Throwable): Unit =
+ rethrowIfMustPropagate(e, fromRead = true)
+
+ private def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new SparkException("A storage filter met a fatal error it must not absorb", internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ if (Executor.isFatalError(e, if (fromRead) fatalErrorDepth - 1 else fatalErrorDepth)) {
+ // The one `Executor.isFatalError` found, which no SparkOutOfMemoryError can precede.
+ val fatal = Iterator.iterate(e)(_.getCause).find(Utils.isFatalError).get
+ if (!corrupt(fatal)) throw out(fatal)
+ }
+ }
+
+ /**
+ * Whether `e` is how `WritableColumnVector.reserve` reports a vector that cannot grow: a plain
+ * `RuntimeException`, caused by an `OutOfMemoryError` or by nothing, for a capacity past what a
+ * vector can hold.
+ */
+ private def isVectorGrowthFailure(e: Throwable): Boolean =
+ e.getClass == classOf[RuntimeException] &&
+ (e.getCause == null || e.getCause.isInstanceOf[OutOfMemoryError])
+
+ /**
+ * Throws a pending task kill, for the reader's loops that can run long without returning to
+ * `FileScanRDD`, which is where a plain read meets one.
+ */
+ def throwIfKilled(): Unit = throwIfKilled(null)
+
+ /**
+ * Throws a pending task kill, wrapped in a checked [[SparkException]] around `surfaced`, the
+ * error the kill surfaced as, if any. Under `ignoreCorruptFiles`, `FileScanRDD` would log any
+ * `RuntimeException` or `IOException` from a reader as a corrupt file and go on to open the
+ * split's remaining files, since it checks for a kill only between batches. The executor reports
+ * the task as killed either way.
+ */
+ private def throwIfKilled(surfaced: Throwable): Unit = {
Review Comment:
Minor, for a follow-up rather than this PR: this wrapper covers a kill that this reader sees at its own checks, but not one that surfaces as an `IOException` from parquet's reads in this reader, such as the `InterruptedIOException` a vectored read or an HDFS socket raises when a speculative attempt is killed with `interruptThread = true`. Phase 1's `readFilteredRowGroup` (`LateMaterializationParquetRecordReader.java:531-532`) and phase 2's (`LateMaterializationParquetRecordReader.java:566` and `586`) let it through as it is, so under `ignoreCorruptFiles`, `FileScanRDD` still logs "Skipped the rest of the content in the corrupted file" and opens the split's next file, which the scaladoc at L313-319 says this prevents. A plain read does the same, so it is not a regression, and the root cause is shared: `DataSourceUtils.shouldIgnoreCorruptFileException` does not ask whether the task was killed. Could a follow-up make it answer false while `TaskContext.get()` is interrupted, the way S
PARK-50483 and SPARK-50545 made the same catch rethrow `BlockMissingException` and `AccessControlException`? That would cover every reader, and the scaladoc here could then speak only of the kills this reader sees itself.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.{SparkContext, SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.Utils
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * saved against a read of the same projection without one. The one cost taken off is a second
+ * key read in a row group that gave splicing up, below. A row group whose filter was given up
+ * reads its key columns twice and counts nothing, so the counters can show a saving on a scan
+ * that read more than it would have with the feature off. Nor do they take off the offset indexes
+ * phase 2 reads for a narrowed row group when no data filter is pushed, which a plain read does not
+ * read. Bytes are the compressed bytes the reader did not ask parquet for. The file system can
+ * still fetch some of them, for example when a vectored read merges nearby ranges.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose non-key columns were never read. Phase 1 read
+ * their key columns, so key bytes are never counted as avoided. In a file that has none of its
+ * key columns, nothing of a skipped row group was read.
+ * - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed data filter's column
+ * index kept.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have asked for over those rows.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is the pages it skipped. A row group that gave splicing up reads its
+ * key columns again in phase 2, which is taken off, down to zero.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose non-key columns the filter kept the reader from reading at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /**
+ * A fresh set of counters, with the labels the SQL UI shows, keyed for the scan, which carries
+ * them without naming any of them.
+ */
+ def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
+ ROW_GROUPS_SKIPPED ->
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ ROWS_EXCLUDED_BY_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ ROWS_EXCLUDED_WITHIN_ROW_GROUP ->
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made. The map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ /**
+ * The predicate this filter evaluates, which the reader resolves once and then runs per row. It
+ * is handed out rather than wrapped, because resolving the `lazy val` costs a volatile read that
+ * a per-row loop would otherwise pay. Codegen-produced predicates can be awkward to serialize
+ * from driver to executor, so it is built on first use on the executor.
+ */
+ @transient lazy val preparedPredicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ if (boundExpression == null) {
+ throw SparkException.internalError("a storage filter must only be evaluated on an " +
+ "executor, and this copy has released the expression it would build its predicate from")
+ }
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets a columnar scan's task reclaim
+ // those bytes for the rest of its life. A row-based scan keeps them anyway, since its closure
+ // captures the scan node, whose storage filters hold the same bytes. Safe because every task
+ // attempt deserializes its own copy from the driver's bytes, and this lazy val is the only
+ // reader of the field.
+ //
+ // It does mean the filter must only ever be evaluated on an executor. Were the driver's copy
+ // to run this, it is the copy serialized to every attempt, and it would go out with nothing
+ // left to build a predicate from.
+ boundExpression = null
+ created
+ }
+}
+
+object ParquetStorageFilter {
+
+ /**
+ * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to
+ * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in
+ * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must
+ * satisfy all of them to survive.
+ *
+ * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all
+ * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a
+ * different matter. These conditions are about the filter being well formed at all.
+ *
+ * Callers that have no storage filters must not call this at all.
+ */
+ def create(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics,
+ maxSplicedRowGroupBytes: Long): ParquetStorageFilter = {
+ require(boundExpressions.nonEmpty,
+ "storage filters must be non-empty; callers with nothing to push must not call create")
+ val expr = boundExpressions.reduce(And)
+
+ // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice
+ // is still one key column) and sorted. Sorting only makes the key row's layout deterministic:
+ // nothing depends on the order, because both `keyColumnIndices` and the remapped references
+ // derive from this list, and the reader pairs each key column with the slot this list names.
+ val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted
+ // These messages name the expression by its node rather than printing it: a prepared bloom
+ // holds its filter as a binary literal, which renders as megabytes of hex.
+ require(originalOrdinals.nonEmpty,
+ s"storage filter ${expr.prettyName} has no bound reference to a key column")
+ require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length),
+ s"storage filter ${expr.prettyName} references ordinals " +
+ s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " +
+ s"fields of ${requestedSchema.catalogString}")
+ val keyFields = originalOrdinals.map(requestedSchema.fields(_))
+ val unsupported = keyFields.filterNot(field => isSupportedKeyType(field.dataType))
+ require(unsupported.isEmpty,
+ "storage filter key columns must have a type the vectorized reader can copy, but " +
+ unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") +
+ " do not; see ParquetStorageFilter.isSupportedKeyType")
+ require(!keyFields.exists(f => f.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME),
+ "a storage filter must not have a key column named " +
+ s"${ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME}, because the reader writes row " +
+ "indexes over that column, so what it reads back depends on how its row group was read")
+
+ val indexMap = originalOrdinals.zipWithIndex.toMap
+ val remapped = expr.transform {
+ case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable)
+ }
+
+ new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes)
+ }
+
+ /**
+ * Whether `dt` is usable as a storage-filter key column type, which is whether the reader's
+ * per-type value copier handles it. [[isSupportedStorageFilter]] asks this for the planner and
+ * [[create]] re-checks it, so `ValueCopier` is only ever asked for a type it has. It is the one
+ * list. See `ValueCopier` for which types are in it and why.
+ */
+ def isSupportedKeyType(dt: DataType): Boolean = ValueCopier.supports(dt)
+
+ /**
+ * Whether the reader can evaluate `expr` as a storage filter, which is what
+ * `ParquetFileFormat.supportsStorageFilter` answers for the planner. Three conditions:
+ *
+ * - the whole conjunct is a `BloomFilterMightContain`, not something with a bloom nested under
+ * an OR or a NOT. The reader evaluates the expression it is given and reads a false as "drop
+ * this row";
+ * - every column it references has a type the reader's value copier can handle. Every reference
+ * is checked, not only the ones under the hash, because [[create]] binds all of them;
+ * - no referenced column is named like the synthetic row-index metadata column. This reader
+ * finds that column by name and writes row indexes over whatever the file holds (SPARK-40059),
+ * so such a column would read back differently depending on how its row group was read, from
+ * the survivor queue where the row group was spliced, from the overwritten vector where it
+ * was not.
+ *
+ * Nothing here asks whether the expression is safe to evaluate on every row. The reader handles
+ * that where it arises, see [[rethrowIfMustPropagate]].
+ */
+ def isSupportedStorageFilter(expr: Expression): Boolean = expr match {
+ case bloom: BloomFilterMightContain =>
+ bloom.references.forall(a => isSupportedKeyType(a.dataType)) &&
+ !bloom.references.exists(a => a.name == ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ case _ => false
+ }
+
+ /**
+ * Rethrows an error the reader met while evaluating a storage filter, if it must not fall back
+ * on it by giving the filter up. The checks run in the executor's own order:
+ * - a fatal error goes out as it is, kill or no kill.
+ * - a pending task kill goes out wrapped, see [[throwIfKilled]]. The error may be the kill's
+ * interrupt surfacing through a UDF that wrapped it. `PythonRunner` reads a kill off the task
+ * context in the same way.
+ * - a fatal error deeper in the cause chain, found by `Executor.isFatalError` as deep as the
+ * executor would look had the post-scan Filter raised `e`, which it raises as it is, goes out
+ * as itself, so the executor finds it at the top whatever wraps it on the way.
+ *
+ * An `InternalError` among those goes out in a checked wrapper. `FileScanRDD` would take a bare
+ * one for a corrupt file under `ignoreCorruptFiles`, where a plain plan raises it from the
+ * post-scan Filter, outside that catch. The executor finds it under the wrapper and the
+ * FAILED_READ_FILE around that at a `spark.executor.killOnFatalError.depth` of 3 or more, the
+ * default being 5.
+ *
+ * Anything else is the filter's own error, which the post-scan Filter raises again for a row it
+ * does evaluate, so the reader falls back on it.
+ */
+ def rethrowIfMustPropagate(e: Throwable): Unit = rethrowIfMustPropagate(e, fromRead = false)
+
+ /**
+ * As [[rethrowIfMustPropagate]], for an error decoding the key pages, with three differences:
+ * - a plain read raises that from the reader, where `FileScanRDD` wraps it in FAILED_READ_FILE
+ * before the executor looks, so a fatal error deeper in the chain has to sit one level
+ * shallower to count.
+ * - an `InternalError` is corrupt data, as `DataSourceUtils.shouldIgnoreCorruptFileException`
+ * takes it.
+ * - a key vector that cannot grow (see [[isVectorGrowthFailure]]) is phase 1's own, since
+ * phase 1 decodes into vectors only this reader allocates. Giving the extra work up is the
+ * answer to that.
+ * The reader falls back on those, and the plain read of the same pages meets the corrupt data
+ * again, or allocates what a plain read allocates.
+ */
+ def rethrowIfMustPropagateFromRead(e: Throwable): Unit =
+ rethrowIfMustPropagate(e, fromRead = true)
+
+ private def rethrowIfMustPropagate(e: Throwable, fromRead: Boolean): Unit = {
+ def corrupt(t: Throwable): Boolean = fromRead && t.isInstanceOf[InternalError]
+ def out(fatal: Throwable): Throwable = fatal match {
+ case internal: InternalError =>
+ new SparkException("A storage filter met a fatal error it must not absorb", internal)
+ case other => other
+ }
+ if (Executor.isFatalError(e, 1) && !corrupt(e)) throw out(e)
+ throwIfKilled(e)
+ if (fromRead && isVectorGrowthFailure(e)) return
+ if (Executor.isFatalError(e, if (fromRead) fatalErrorDepth - 1 else fatalErrorDepth)) {
+ // The one `Executor.isFatalError` found, which no SparkOutOfMemoryError can precede.
+ val fatal = Iterator.iterate(e)(_.getCause).find(Utils.isFatalError).get
+ if (!corrupt(fatal)) throw out(fatal)
+ }
+ }
+
+ /**
+ * Whether `e` is how `WritableColumnVector.reserve` reports a vector that cannot grow: a plain
+ * `RuntimeException`, caused by an `OutOfMemoryError` or by nothing, for a capacity past what a
+ * vector can hold.
+ */
+ private def isVectorGrowthFailure(e: Throwable): Boolean =
+ e.getClass == classOf[RuntimeException] &&
+ (e.getCause == null || e.getCause.isInstanceOf[OutOfMemoryError])
+
+ /**
+ * Throws a pending task kill, for the reader's loops that can run long without returning to
+ * `FileScanRDD`, which is where a plain read meets one.
+ */
+ def throwIfKilled(): Unit = throwIfKilled(null)
+
+ /**
+ * Throws a pending task kill, wrapped in a checked [[SparkException]] around `surfaced`, the
+ * error the kill surfaced as, if any. Under `ignoreCorruptFiles`, `FileScanRDD` would log any
+ * `RuntimeException` or `IOException` from a reader as a corrupt file and go on to open the
+ * split's remaining files, since it checks for a kill only between batches. The executor reports
+ * the task as killed either way.
+ */
+ private def throwIfKilled(surfaced: Throwable): Unit = {
+ val context = TaskContext.get()
+ if (context != null && context.isInterrupted()) {
+ val cause =
+ if (surfaced != null) surfaced else new TaskKilledException(context.getKillReason().get)
+ throw new SparkException("A storage filter met a task kill it must not absorb", cause)
+ }
+ }
+
+ /**
+ * Throws `SparkException.internalError(message, cause)`, for a broken invariant of this reader.
+ * The exception is checked, so `FileScanRDD` cannot read it as a corrupt file under
+ * `ignoreCorruptFiles` and skip the rest of the file silently. Typed as returning a
+ * `RuntimeException` only so that Java code can write `throw internalError(...)` without
+ * declaring a checked exception. It never returns.
+ */
+ def internalError(message: String, cause: Throwable): RuntimeException =
Review Comment:
Nit: no caller passes a cause. All six, `LateMaterializationParquetRecordReader.java:376`, `413`, `421` and `694`, `ParquetReadState.java:111` and `ValueCopier.java:75`, use the one-argument overload, which passes `null` here, and `SparkException.internalError(msg)` builds the same exception. Could this be one `def internalError(message: String): RuntimeException = throw SparkException.internalError(message)`?
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1051 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer has passed
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * phase 2 reads its key columns again with the rest. If the survivors' row ranges alone pass it,
+ * the filter is given up for that row group too.
+ *
+ * <p>A whole file is read without splicing in these cases:
+ * <ul>
+ * <li>every key column is missing from it under schema evolution. The predicate is then constant
+ * over the file and is evaluated once, against the constants the reader materializes for
+ * those columns. True keeps the file unfiltered, false skips it;</li>
+ * <li>the file has none of the projected non-key columns, or none is projected. Phase 2 would
+ * have nothing to read, so the reader declines the filter and reads the file plainly;</li>
+ * <li>{@code parquet.filter.columnindex.enabled} is false, so its page index is not used. The
+ * filter then only skips row groups it rejects whole.</li>
+ * </ul>
+ *
+ * <p>A row group whose columns do not all carry an offset index gets the same as the last case,
+ * and so does one whose offset index parquet fails to read.
+ *
+ * <p>{@link #resultBatch()} is one batch object for the whole read, as for the plain reader, but a
+ * spliced key slot holds a different vector in each batch, and {@link #nextBatch()} frees the
+ * previous one. A caller must fetch {@code batch.column(i)} again after every {@code nextBatch()}.
+ *
+ * <p>What {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}
+ * guarantee is asserted here, since a violation is a planner bug.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A byte child sized up
+ * front is charged by the same factor when it is sized, see {@link #allocateKeyAccumulators}.
+ * A fixed-width key needs no factor, since its accumulator is allocated at capacity and never
+ * grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * What one surviving row range costs to hold, which is the half of the budget a row group
+ * buffering nothing still pays. Scattered survivors make one range each, and phase 2 needs the
+ * whole set to select its pages. One set is held, which every column reader of the row group
+ * walks with a cursor of its own. The figure is parquet's {@code RowRanges.Range}, two longs,
+ * plus its object header and a list slot. That is what is held. At the end of phase 1 the
+ * builder's list can have up to half as many slots again, and {@code build()} copies it, so the
+ * peak is somewhat higher.
+ */
+ private static final long ROW_RANGE_BYTES = 40L;
+
+ /**
+ * The byte child a variable-length vector is allocated with, per row, which is
+ * `WritableColumnVector.DEFAULT_ARRAY_LENGTH`. `keyFixedBytesPerRow` already charges it.
+ */
+ private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+
+ /**
+ * The filter this reader applies. {@link #keyColumns} is null for a file it is not applied to row
+ * by row, and such a file is read the way the plain reader reads it.
+ */
+ private final ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, set by {@link #initializeLateMaterialization()}.
+ *
+ * <p>{@link #fileReader} drives all three phases, and its requested schema is switched per phase.
+ * Phase 0 reads the whole projection, phase 1 the key columns, and phase 2 the non-key columns
+ * while splicing or the whole projection otherwise.
+ *
+ * <p>Held as leaf-column lists, since that is what the reader and the byte metrics take and
+ * {@code MessageType.getColumns()} rebuilds its list per call. The leaf paths of the whole
+ * projection and of the non-key columns are resolved once, for the footer walks that run per row
+ * group. The key columns need none, since phase 1 always reads them and no walk measures them
+ * apart.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /** Answers from the footer whether a row group can be read in part, and what that transfers. */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet does not
+ * close the stores {@code readFilteredRowGroup} hands out.
+ */
+ private PageReadStore dataPages;
+
+ /** The messages {@link #reportOnce} has already reported. */
+ private final Set<String> reportedMessages = new HashSet<>();
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} is false, so this file's page index must not
+ * be used. That turns off phase 0 and every partial read. A wrong index would pair a row's key
+ * with another row's values, which the post-scan Filter cannot catch. Parquet's partial read does
+ * not check the conf itself. Read off {@link #readOptions}, so it agrees with the row count
+ * {@link #fileReader} reported.
+ */
+ private boolean pageIndexDisabled;
+
+ /** Whether the reader has reported an offset index parquet could not read. */
+ private boolean reportedUnreadableOffsetIndex;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts as whether the row group can be
+ * read in part, and {@link #abandonSplicing} clears it. False means phase 2 read every projected
+ * column, so the emit path takes the key columns from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /**
+ * The key columns this file has, in the filter's key order, with what init resolves about each.
+ * Null when the filter is not applied to this file row by row.
+ */
+ private KeyColumn[] keyColumns;
+ /** Whether this file has none of the key columns, decided in {@link #initBatch}. */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it decoded, and the emit path splices
+ * them into the output batch. That saves phase 2 a second read of the key columns, which parquet
+ * does not cache, so it goes back to the filesystem.
+ *
+ * <p>{@link #survivorBatches} holds one array of survivor vectors per batch to emit, indexed like
+ * {@link #keyColumns}, and {@link #currentKeyAccumulators} the array still filling. Together they
+ * hold every survivor of one row group at once, outside any MemoryConsumer.
+ */
+ private final ArrayDeque<WritableColumnVector[]> survivorBatches = new ArrayDeque<>();
+ /** The survivor vectors still filling, or null when there are none. */
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}. */
+ private int currentKeyAccumulatorRowCount;
+ /**
+ * Per key column, the part of what {@link #currentKeyAccumulators}' byte child was charged when
+ * it was sized that its values have not used up yet. Always zero for a fixed-width key.
+ */
+ private long[] prepaidValueBytes;
+ /**
+ * The survivor vectors the current batch is built on. This reader owns them until the next batch
+ * releases them, so a consumer that stops between batches, such as a limit or the end of the
+ * task, leaves them for {@link #close()} to free.
+ */
+ private WritableColumnVector[] publishedKeyVectors;
+ /** {@link #initBatch}'s vectors, which this reader closes. */
+ private ColumnVector[] persistentBatchColumns;
+ /** The array the emitted batch is built over, whose key slots each batch rewrites. */
+ private ColumnVector[] spliceBatchColumns;
+
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ /**
+ * Releases what this reader allocated, then what the base owns. Chained through finally blocks,
+ * so one failing close does not leak the rest.
+ */
+ @Override
+ public void close() throws IOException {
+ try {
+ try {
+ if (persistentBatchColumns != null) {
+ // Through the persistent vectors rather than the emitted batch, whose key slots may hold
+ // survivor vectors the splicing state closes instead. Super then has no batch to close.
+ columnarBatch = null;
+ closeAll(persistentBatchColumns);
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ }
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ try {
+ closeDataPages();
+ } finally {
+ super.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Builds the batch as the plain reader does and takes it over for splicing. A file with none of
+ * the key columns is decided here, because the constant vectors the predicate reads for those
+ * columns exist only once super has built them.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built. For a file the filter is applied to row by row, it
+ * then hands out a batch of its own over a copy of that array, whose key slots the emit path
+ * rewrites in place, and allocates phase 1's key vectors. {@link ColumnarBatch} holds the array
+ * by reference, its staging row included, so rewriting a slot is what publishes it.
+ *
+ * <p>Every slot got a vector, key columns included. A spliced row group leaves a key slot's
+ * vector unused, and a row group that gave splicing up reads into it.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ allocateKeyScratch();
+ }
+
+ /**
+ * Answers the filter for a file with none of its key columns, which makes the predicate constant
+ * over the file.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ // Resolved outside the `try`, so a broken invariant behind it fails rather than falls back.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ boolean keep;
+ try {
+ keep = predicate.eval(keyRow.getRow(0));
+ } catch (Throwable t) {
+ giveUpOnError(t, FILE_GIVEN_UP_ON_EVALUATION);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ }
+
+ /**
+ * Releases the survivor vectors the previous batch was built on, reads the next batch the way the
+ * plain reader does, and then points the emitted batch's key slots at this batch's vectors. A
+ * spliced row group takes them from the head of the survivor queue, and any other row group from
+ * the persistent vectors, which is what a row group following a spliced one needs. Non-key slots
+ * keep {@link #initBatch}'s vectors for the reader's life.
+ *
+ * <p>The row-index column the base fills is never a key column, since
+ * {@code ParquetStorageFilter.create} rejects its name. So it always lands in a persistent
+ * vector.
+ */
+ @Override
+ public boolean nextBatch() throws IOException {
+ closeAll(publishedKeyVectors);
+ publishedKeyVectors = null;
+ if (!super.nextBatch()) return false;
+ if (keyColumns != null) {
+ if (spliceCurrentRowGroup) {
+ publishedKeyVectors = survivorBatches.pollFirst();
+ if (publishedKeyVectors == null) {
+ // Unreachable, since the queue holds exactly the survivors phase 1 accumulated and the
+ // emit loop is driven by that same count. Without this, the key slots would point at the
+ // persistent vectors, which a spliced row group resets but never reads into, so the batch
+ // would pair stale key values with this row group's rows as if they were real ones.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter survivor queue of row group %d in %s ran out with %d rows still "
+ + "to emit", nextBlockIndex - 1, fileReader.getFile(), columnarBatch.numRows()));
+ }
+ }
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ publishedKeyVectors != null ? publishedKeyVectors[k] : persistentBatchColumns[slot];
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // A row-group reader was handed in with no file reader behind it, so the phases cannot run.
+ // The file is read the plain way.
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ pageIndexDisabled = !readOptions.useColumnIndexFilter();
+ // Resolve each key column's top-level ParquetColumn and keep the ones this file has. A key
+ // column can be missing under schema evolution.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw ParquetStorageFilter.internalError(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw ParquetStorageFilter.internalError(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ keyTopLevelNames.add(column.descriptor().get().getPath()[0]);
+ }
+ }
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyPositions.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. `initBatch`
+ // decides it, once the vectors it evaluates against exist.
+ allKeysMissing = true;
+ return;
+ }
+
+ // The non-key columns phase 2 reads. If this file has none of them, the filter saves nothing,
+ // since phase 1 already reads every column the file has for every row. The reader then
+ // declines it. The planner already declines a projection of key columns alone, so this catches
+ // a file missing every projected non-key leaf, which also covers the row-index column and a
+ // struct none of whose requested fields the file has.
+ List<ColumnDescriptor> nonKey = requestedColumns.stream()
+ .filter(column -> !keyTopLevelNames.contains(column.getPath()[0]))
+ .toList();
+ if (nonKey.stream().noneMatch(column -> fileSchema.containsPath(column.getPath()))) return;
+ nonKeyColumns = nonKey;
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ KeyColumn[] keys = new KeyColumn[presentKeyPositions.size()];
+ StructField[] fields = sparkRequestedSchema.fields();
+ int fixedBytesPerRow = 0;
+ for (int i = 0; i < keys.length; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ ParquetColumn column = parquetColumn.children().apply(batchSlot);
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keys[i] = new KeyColumn(batchSlot, rowPosition, column.descriptor().get(),
+ column.required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ // A decimal of up to 9 digits is held as an int, though its default size is a long's.
+ int valueWidth = type instanceof DecimalType && DecimalType.is32BitDecimalType(type)
+ ? 4 : type.defaultSize();
+ fixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + valueWidth;
+ }
+ keyFixedBytesPerRow = fixedBytesPerRow;
+ keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ prepaidValueBytes = new long[keys.length];
+ // Last, since a non-null `keyColumns` is what says the filter is applied to this file.
+ keyColumns = keys;
+ }
+
+ /**
+ * Loads the next row group with rows left to emit, through the three phases of the class javadoc.
+ * Phase 2 reads the non-key columns over the survivors while splicing, the whole projection over
+ * the survivors once splicing is given up, and the whole projection over
+ * {@code pushedFilterRanges} once the filter is. Returns false when the row groups run out, which
+ * ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ if (keyColumns == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so a row group the filter empties,
+ // or the end of the file, does not keep the previous one's pages alive.
+ closeDataPages();
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ while (nextBlockIndex < blocks.size()) {
+ // A plain read goes back to `FileScanRDD`, which checks for a task kill, after every batch.
+ // This loop can pass any number of row groups the filter empties without emitting one.
+ ParquetStorageFilter.throwIfKilled();
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+
+ // Phase 0: rows the pushed data filter allows, by column index. The full projection goes back
+ // on first, because `getRowRanges` builds an unseen block's store over the current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // An empty block, or one the pushed data filter rejects whole. Not this filter's skip, so
+ // no storage-filter metric counts it.
+ continue;
+ }
+
+ // Whether phase 2 can read part of this row group. That needs an offset index for every
+ // column of the block's column index store, and parquet empties the store if one column
+ // lacks it. With a data filter pushed, the store is built at initialize over the whole
+ // projection, so every projected column is checked. Files written before parquet-mr 1.11,
+ // or by pyarrow's `write_table` by default, have no offset index at all. With no data filter
+ // pushed the check is stricter than needed, since the store would cover only the non-key
+ // columns. False means nothing is buffered and the filter can only skip the row group whole.
+ // Phase 2 then reads the whole projection over `pushedFilterRanges`. Parquet narrows those
+ // only by an index it could read, so that read cannot throw.
+ boolean canNarrowRowGroup =
+ !pageIndexDisabled && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1 reads the key columns over `pushedFilterRanges` and evaluates the filter per row.
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ RowRanges survivors;
+ // Closed at the end of the phase that reads it, since parquet closes no store it hands out.
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ requireRowCount(keyPages, baselineRows, blockIdx);
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null means phase 1 gave the filter up for this row group, and phase 2 then reads every row
+ // the pushed filter kept, as a plain read would. Phase 2 can also give it up, in the catch
+ // below.
+ boolean filterGivenUp = survivors == null;
+ if (filterGivenUp) abandonSplicing();
+ RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
+ long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
+ // The byte metrics' baseline, the non-key bytes a plain read asks for over the rows the
+ // pushed filter kept. Not walked for a row group whose filter was given up.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (finalRowCount == 0) {
+ // The filter rejected every row, so the row group is skipped whole.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // The footer check above catches every file written without offset indexes. This catch is
+ // the backstop for an offset index the footer lists but parquet fails to read, which empties
+ // that block's store, and only that block's, since parquet keeps one per block. The row group
+ // is then read again over `pushedFilterRanges`, which that empty store has already widened to
+ // the whole block, so the retry cannot throw. The next row group is narrowed as usual.
+ //
+ // `MissingOffsetIndexException` is parquet's internal API, as are the `ColumnIndexStore` and
+ // `OffsetIndex` that `ParquetBlockAccounting` reads. Revisit both on a parquet upgrade. If
+ // parquet stops throwing this one as it is, the damaged-index test in
+ // `ParquetStorageFilterSuite` fails.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // Parquet turns any IOException reading an offset index into this, a kill's interrupt
+ // included, which must not be taken for a missing index.
+ ParquetStorageFilter.throwIfKilled();
+ if (!reportedUnreadableOffsetIndex) {
+ reportedUnreadableOffsetIndex = true;
+ // The message only, since parquet has already logged the failed index read with its
+ // stack trace.
+ LOG.warn("Reading a row group of {} without page-level storage filtering, because "
+ + "parquet could not read the offset index of a column the read needs ({}). The "
+ + "other row groups are filtered as usual. Reported once per split",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ }
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ requireRowCount(dataPages, finalRowCount, blockIdx);
+ // A row group whose filter was given up reports nothing (see `StorageFilterMetrics`).
+ if (!filterGivenUp) {
+ // Over the columns phase 2 read, which include the key columns again once splicing is
+ // given up. A spliced row group that pruned nothing read exactly the baseline.
+ long phase2Bytes = spliceCurrentRowGroup && finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(blockIdx,
+ spliceCurrentRowGroup ? nonKeyPaths : requestedPaths, finalRanges, finalRowCount);
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ m.rowsExcludedWithinRowGroup().add(baselineRows - finalRowCount);
+ }
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ installRowGroup(dataPages, readerRangesFor(finalRanges, finalRowCount, blockRowCount));
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queue at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ if (!spliceCurrentRowGroup) return false;
+ for (KeyColumn key : keyColumns) {
+ if (key.batchSlot() == slot) return true;
+ }
+ return false;
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group the pushed data filter left rows in counts as skipped, with every projected byte of those
+ * rows as avoided. Those are all non-key bytes, since the file has no key column.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ List<BlockMetaData> blocks = fileReader.getRowGroups();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blocks.get(blockIdx).getRowCount());
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ if (blockRowCount == 0) {
+ // parquet-mr never writes an empty block, and the plain read path skips one too. For one,
+ // `RowRanges.createSingle(0)` below would build Range(0, -1) and trip its `from <= to`
+ // assertion.
+ return RowRanges.EMPTY;
+ }
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, which is required rather than cheaper. Parquet reads the block
+ * whole then and hands back a store with no row indexes, and a column reader refuses ranges it
+ * cannot find in the pages by them (see {@code ParquetReadState.forRead}).
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * Fails unless a page store holds as many rows as were asked for, which a null store never does,
+ * since every caller asks for some. Parquet returns null only for an empty block or empty
+ * ranges, and otherwise exactly the rows asked for. The column readers take their rows from the
+ * ranges and the emit loop takes its count from the store, so a store that held a different
+ * number would leave a batch's tail stale, or splice survivors into the next row group's rows,
+ * with no error.
+ */
+ private static void requireRowCount(PageReadStore pages, long expected, int blockIdx) {
+ long actual = pages == null ? 0L : pages.getRowCount();
+ if (actual != expected) {
+ throw ParquetStorageFilter.internalError(String.format(
+ "Row group %d was read as %d rows where %d were asked for", blockIdx, actual, expected));
+ }
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}.
+ *
+ * <p>While splicing, each survivor's key values are appended to the accumulators for the emit
+ * path to splice, until the buffer passes its cap.
+ *
+ * <p>Returns null once the reader gives the filter up for this row group, which happens on:
+ * <ul>
+ * <li>the first survivor of a row group that cannot be narrowed, since whether anything
+ * survives was all that was left to learn;</li>
+ * <li>the surviving ranges passing the budget;</li>
+ * <li>an error decoding the key pages or applying the filter.</li>
+ * </ul>
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group keeps, which the budget below weighs, is the bytes buffered for splicing
+ // and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long rangeBytes = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ // Per chunk, since this decodes and evaluates a whole row group before its first batch.
+ ParquetStorageFilter.throwIfKilled();
+ int num = (int) Math.min((long) capacity, remaining);
+ try {
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ } catch (Throwable t) {
+ // A corrupt page shows here, since parquet reads a chunk's bytes up front but decodes a
+ // page only when it is read. That includes a codec that reports corruption as an
+ // `InternalError`, as hadoop-lzo does. Giving the row group up hands it to phase 2 whole,
+ // which reads the same pages in the same batches as a plain scan. So the page fails at the
+ // same batch, after the same rows, and `ignoreCorruptFiles` keeps what it keeps of a plain
+ // read.
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(t);
+ reportOnce(t, ROW_GROUP_GIVEN_UP_ON_DECODING);
+ return null;
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ boolean survives;
+ try {
+ survives = predicate.eval(keyScratchBatch.getRow(r));
+ } catch (Throwable t) {
+ // Fall back on anything but a task kill or a fatal error. The predicate runs here without
+ // the conjuncts before it in the plan, so it can throw on a row a plain scan never
+ // evaluates it on (see `ParquetStorageFilter.isSupportedStorageFilter`). Giving the
+ // filter up returns every row the pushed filter kept, and the post-scan Filter decides
+ // them in its own order, raising the same error for a row it does evaluate.
+ //
+ // This is not narrowed by type. A built-in expression can throw a plain JDK exception
+ // (`timestamp_seconds` of a decimal) or an `AssertionError`, and a Hive UDF throws a
+ // checked SparkException. Rethrowing would fail a query a plain read answers, or drop
+ // the rest of the file silently under `ignoreCorruptFiles`.
+ giveUpOnError(t, ROW_GROUP_GIVEN_UP_ON_EVALUATION);
+ return null;
+ }
+ if (!survives) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so one survivor settles it.
+ // Stop decoding. A row group with no survivor still gets empty ranges below and is
+ // skipped.
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) {
+ try {
+ if (currentKeyAccumulators == null) {
+ splicedBytes += allocateKeyAccumulators(
+ cap - splicedBytes - rangeBytes - keyFixedBytesPerRow);
+ }
+ splicedBytes += appendSurvivorRowToAccumulators(r);
+ } catch (RuntimeException e) {
+ // The buffer is this reader's own and optional, so failing to fill it gives splicing up
+ // rather than the filter, and the memory goes back at once. What fails here:
+ // - a survivor vector that cannot grow;
+ // - a key value the predicate did not read on this row, such as `b` in
+ // `coalesce(a, b)`, since the copy decodes every key;
+ // - a bug of ours, which only the WARN then shows.
+ // Phase 2 reads the key columns the way a plain read does, so a value that cannot be
+ // decoded fails the read only where a plain read fails.
+ reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
+ abandonSplicing();
+ splicedBytes = 0L;
+ }
+ }
+ // Both parts of what this row group keeps grow per survivor, so they are weighed together.
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper step first. Releasing the buffer is a no-op once splicing is given up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ // The ranges being built would be thrown away, so stop evaluating the rest.
+ if (rangeBytes > cap) return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // A partly filled set of accumulators goes to the queue as this row group's last batch.
+ if (currentKeyAccumulators != null) pushAccumulators();
+ return finalRangesBuilder.build();
+ }
+
+ /** Allocates phase 1's key vectors and the row the predicate evaluates over them. */
+ private void allocateKeyScratch() {
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // Missing key columns keep their batch vectors (see `keyVectorsFromBatch`), which the emitted
+ // batch owns.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors from the emitted batch, in the order the predicate's row reads them.
+ * A key column this file does not have keeps its batch vector, which holds what the scan returns
+ * for it (its existence DEFAULT or null). So the filter sees exactly the values the rows show.
+ * Phase 1 replaces the others with its own vectors. Call only after {@link #takeOverBatch}.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = persistentBatchColumns[keyIndices[p]];
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates a set of accumulators, {@link #capacity} rows each, and returns the charge for the
+ * byte children it sized up front. The rest of the set is charged per row as survivors arrive.
+ * A variable-length key's byte child is reserved for what the set queued before it held, where
+ * that is more than its default allocation, so it does not grow one doubling at a time for every
+ * capacity's worth of survivors. {@code reserve} doubles what it is asked for, so with its null
+ * bytes the child takes the factor's four bytes per value byte. That happens only when the
+ * charge fits within {@code headroom}, what the cap leaves after this survivor's own fixed part.
+ * The memory is taken before any value arrives, so it is charged now, as if those values had
+ * arrived, and the values written into it use that charge up before they add to it.
+ */
+ private long allocateKeyAccumulators(long headroom) {
+ WritableColumnVector[] previous = survivorBatches.peekLast();
+ int[] sizeTo = new int[keyColumns.length];
+ long charge = 0L;
+ for (int i = 0; i < keyColumns.length; i++) {
+ // Only a child the default allocation would not already hold gains from being sized.
+ int valueBytes = previous != null && keyColumns[i].variableLength()
+ ? previous[i].arrayData().getElementsAppended() : 0;
+ if (valueBytes > capacity * DEFAULT_CHILD_BYTES_PER_ROW) {
+ sizeTo[i] = valueBytes;
+ charge += VARIABLE_LENGTH_BYTES_FACTOR * valueBytes;
+ }
+ }
+ // The memory is taken before the check after this survivor, so only when it fits what is left
+ // of the cap. A set that is not sized up front grows as the first one does, charged value by
+ // value, which also keeps a nearly empty last set from being charged as a full one.
+ boolean sizeUpFront = charge <= headroom;
Review Comment:
Minor: a later set is sized to the previous set's value bytes whenever the charge fits, however few rows can still reach it, so a row group's last set is allocated and charged as a full one. The new test shows what that does to the cap (`ParquetStorageFilterSuite.scala:1021-1064`): with six 1000-byte keys at capacity 4, the second set is sized for 4000 value bytes and charged 16000 for two survivors, so caps of 28000 and 32120 keep splicing, 32125 to 32141 give it up at the sixth survivor and read the key column twice, and 40000 keeps it again. A larger cap reads more. At capacity 4096 with 1000-byte keys, such a last set takes about 16.4MB, however few rows land in it. Could the sizing scale with the rows left, `remaining - r` passed from L796, e.g. `valueBytes * min(capacity, rowsLeft) / capacity` and only when that is past the default allocation, and could the headroom also leave room for the fixed part of those rows? Some caps of that test would need re-pinning.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether a log line containing `text` was logged.
+ private def withLogged[T](text: String)(body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains(text))
+ (result.get, logged)
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) =
+ withLogged("without page-level storage filtering")(body)
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def readWith(key: Expression): SparkException = intercept[SparkException] {
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ readAll(path, filter)._2.close()
+ }
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException = {
+ TaskContext.setTaskContext(TaskContext.empty())
+ try readWith(key) finally TaskContext.unset()
+ }
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_))),
+ classOf[SparkException])
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ val key = OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped")))
+ readAll(path, createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested))._2
+ .close()
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in a checked wrapper, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ assertPropagates(readWith(OnKey(bound, at = 7L, error = new InternalError(_))),
+ classOf[InternalError])
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val assertingKey = OnKey(bound, at = 7L, error = new AssertionError(_))
+ val (result, reader) =
+ readAll(path, createFilter(Seq(GreaterThanOrEqual(assertingKey, Literal(10L))), requested))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagate(under(depth - 1, oom))
+ } eq oom, "the deepest level the executor sees counts")
+ ParquetStorageFilter.rethrowIfMustPropagate(under(depth, oom))
+
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(depth - 2, oom))
+ } eq oom, "a decoding error's chain counts one level less deep")
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(depth - 1, oom))
+
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(1, new InternalError("codec")))
+ // One from an expression goes out in a checked wrapper, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file, and the executor finds it under it.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[SparkException](ParquetStorageFilter.rethrowIfMustPropagate(e))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ assert(Executor.isFatalError(thrown, depth), s"got $thrown")
+ }
+ ParquetStorageFilter.rethrowIfMustPropagate(new AssertionError("not fatal"))
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM.
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(growth)
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagate(growth)
+ }.getMessage == "reserve")
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.setTaskContext(context)
Review Comment:
Nit: `TaskContext.withTaskContext(context) { ... }` (`TaskContext.scala:53-60`) sets the context, runs the body and unsets it in a `finally`, as this block and the ones at L583-584 and L688-696 do by hand. `ValueStateSuite` in this module already uses it. Could these three use it?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether a log line containing `text` was logged.
+ private def withLogged[T](text: String)(body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains(text))
+ (result.get, logged)
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) =
+ withLogged("without page-level storage filtering")(body)
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def readWith(key: Expression): SparkException = intercept[SparkException] {
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ readAll(path, filter)._2.close()
+ }
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException = {
+ TaskContext.setTaskContext(TaskContext.empty())
+ try readWith(key) finally TaskContext.unset()
+ }
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_))),
+ classOf[SparkException])
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ val key = OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped")))
+ readAll(path, createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested))._2
+ .close()
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in a checked wrapper, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ assertPropagates(readWith(OnKey(bound, at = 7L, error = new InternalError(_))),
+ classOf[InternalError])
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val assertingKey = OnKey(bound, at = 7L, error = new AssertionError(_))
+ val (result, reader) =
+ readAll(path, createFilter(Seq(GreaterThanOrEqual(assertingKey, Literal(10L))), requested))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagate(under(depth - 1, oom))
+ } eq oom, "the deepest level the executor sees counts")
+ ParquetStorageFilter.rethrowIfMustPropagate(under(depth, oom))
+
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(depth - 2, oom))
+ } eq oom, "a decoding error's chain counts one level less deep")
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(depth - 1, oom))
+
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(1, new InternalError("codec")))
+ // One from an expression goes out in a checked wrapper, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file, and the executor finds it under it.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[SparkException](ParquetStorageFilter.rethrowIfMustPropagate(e))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ assert(Executor.isFatalError(thrown, depth), s"got $thrown")
+ }
+ ParquetStorageFilter.rethrowIfMustPropagate(new AssertionError("not fatal"))
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM.
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(growth)
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagate(growth)
+ }.getMessage == "reserve")
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.setTaskContext(context)
+ try {
+ assert(intercept[OutOfMemoryError](ParquetStorageFilter.rethrowIfMustPropagate(oom)) eq oom)
+ val killed = intercept[SparkException] {
+ ParquetStorageFilter.rethrowIfMustPropagate(under(1, oom))
+ }
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ } finally {
+ TaskContext.unset()
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ TaskContext.setTaskContext(TaskContext.empty())
+ val thrown = try {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ } finally {
+ TaskContext.unset()
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty)
+ .getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ val (_, sevenUncapped, _) = run("64MB", 394L)
+ val (_, sevenAtCharge, _) = run("103b", 394L)
+ val (_, sevenUnderCharge, _) = run("102b", 394L)
+ assert(sevenAtCharge == sevenUncapped,
+ s"a cap of exactly the charge must keep splicing; got $sevenAtCharge against " +
+ sevenUncapped)
+ assert(sevenUnderCharge > sevenUncapped,
+ s"a cap one byte under it must give splicing up; got $sevenUnderCharge against " +
+ sevenUncapped)
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a later accumulator set is sized up front only when its charge fits under the cap") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, which takes that memory before any value arrives, so it is charged then. Six survivors
+ // of a 1000-byte key over a capacity of 4 fill one set of four, charged about 16KB, and sizing
+ // the next set would charge as much again for two survivors. Under a cap of 28000 bytes that
+ // does not fit, so the set grows value by value instead, and the row group stays spliced, which
+ // reading the key column only once shows. The sizing has to fit after the fifth survivor's own
+ // fixed 17 bytes too. A cap of 32120 leaves 15995 for it, so the set grows and stays spliced.
+ // Under a cap that fits it, the second set is sized and charged once. A cap of 32130 holds
+ // the 32125 bytes of the fifth survivor and gives splicing up at the sixth, at 32142. A cap
+ // of 40000 keeps splicing. A missing or a double charge would each get one of these wrong.
+ // And the keys that land in the second set must come back right either way.
+ withTempDir { dir =>
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), REPEAT('x', 996)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val fileSchema = kvSchema(keyType = StringType)
+ val storageFilters = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, fileSchema, storageFilters, maxSplicedBytes, batchSize = 4)
+
+ val (uncappedRows, uncappedBytes) = run("64MB")
+ val (cappedRows, cappedBytes) = run("28000b")
+ assert(uncappedRows == 6 && cappedRows == 6,
+ s"the filter keeps keys 15..20 either way; got $uncappedRows and $cappedRows")
+ assert(cappedBytes == uncappedBytes,
+ "a set whose sizing does not fit must grow instead, so the row group stays spliced and " +
+ s"reads the key column once; capped=$cappedBytes uncapped=$uncappedBytes")
+ val (_, noRoomBytes) = run("32120b")
+ assert(noRoomBytes == uncappedBytes,
+ "a sizing that fits only without this survivor's own charge must not happen, so a cap of " +
+ s"32120 bytes keeps splicing; got $noRoomBytes against $uncappedBytes")
+ val (_, justUnderBytes) = run("32130b")
+ assert(justUnderBytes > uncappedBytes,
+ "the sizing must be charged when the set is sized, which passes a cap of 32130 bytes; " +
+ s"got $justUnderBytes against $uncappedBytes")
+ val (_, roomyBytes) = run("40000b")
+ assert(roomyBytes == uncappedBytes,
+ "and charged once, so a cap of 40000 bytes keeps splicing; " +
+ s"got $roomyBytes against $uncappedBytes")
+
+ val expected = (15 to 20).map(id => (f"$id%04d" + "x" * 996, id.toString))
+ Seq(Long.MaxValue, 28000L).foreach { cap =>
+ val (pairs, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(storageFilters, fileSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString),
+ capacity = 4)
+ try {
+ assert(pairs == expected,
+ s"every survivor must come back with its own key under cap $cap; got " +
+ pairs.map(_._2))
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ val outDir = new File(dir, "damaged").getAbsolutePath
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ rows.toDF("k", "v").repartition(1).write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 64L * 1024 * 1024)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .option(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT, 100L)
+ .parquet(outDir)
+ val path = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ .head.getAbsolutePath
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(false, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(false, 16, storageFilter)
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ (read.toSeq, error)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError) = readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ }
+ }
+
+ test("a key value the filter does not read and cannot be decoded gives splicing up") {
+ // The filter need not read every key on every row, as `coalesce(a, b)` reads `b` only where
+ // `a` is null. The copy into the survivor buffer reads every key, so it can meet a value no
+ // plain read decodes. A dictionary-encoded column decodes a value only when it is read, so a
+ // dictionary that holds fewer values than its ids point at is such a value. Copying it gives
+ // splicing up for the row group, and phase 2 reads the key columns the way a plain read does.
+ // So the read answers where a plain one does, and the filter still applies.
+ withTempDir { dir =>
+ // Two copies of each `b`, so parquet keeps its dictionary. Cut to 64 values, it leaves `b`
+ // undecodable from row 128 on. `a` is null only before that, so the filter never reads a
+ // damaged `b`.
+ val rows = (0 until 200).map { i =>
+ (if (i < 100 && i % 5 == 0) null else s"a_$i", s"b_${i / 2}", i)
+ }
+ val outDir = new File(dir, "dictionary").getAbsolutePath
Review Comment:
Nit: this and two other new tests (L1476, L2682) write their file and pick it with `listFiles(...).head` instead of calling `writeSingleParquetFile` (L77-100), which asserts that exactly one file was written. This one can call it as it is, with parquet's default row-group size and `dictionary = true`, and an `options: Map[String, String] = Map.empty` applied last, as `ParquetColumnIndexSuite.checkUnalignedPages` does with `extraOptions`, would take the other two and the older copy at L1319. L1480 could then use `ParquetOutputFormat.WRITER_VERSION` instead of the string key.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,3138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.Locale
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.Util
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, TaskKilledException}
+import org.apache.spark.executor.Executor
+import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // 400 rows whose values sort with their keys, the fixture of the page-level tests.
+ private val rows400 = (1L to 400L).map(i => (i, f"v_$i%04d"))
+
+ // `rows400` in one row group of 100-row pages.
+ private def writePagedFile(dir: File): String =
+ writeParquetFile(dir, rows400, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = drain(readerFn(file)) { it =>
+ it.map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ }
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // Consumes what a reader builder returned for one file through `consume`, and closes the reader
+ // behind it whatever happens. Exhausting the iterator closes it too, but a failure part way
+ // through, which is what these tests look for, would otherwise leave it and its stream open.
+ private def drain[T](iter: Iterator[_])(consume: Iterator[Object] => T): T = {
+ try {
+ consume(iter.asInstanceOf[Iterator[Object]])
+ } finally {
+ iter match {
+ case closeable: java.io.Closeable => closeable.close()
+ case _ =>
+ }
+ }
+ }
+
+ // A batch reader for a file of `schema`, built through `ParquetFileFormat` the way a scan with
+ // storage filters builds one.
+ private def formatReader(
+ schema: StructType,
+ pushed: Seq[sources.Filter],
+ storageFilters: Seq[Expression],
+ hadoopConf: Configuration = spark.sessionState.newHadoopConf(),
+ metrics: Map[String, SQLMetric] = metricMap()): PartitionedFile => Iterator[InternalRow] =
+ new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics)
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+
+ // Reads `path` whole through the format under a splice cap and a batch size, and returns the rows
+ // it emitted with the bytes it transferred.
+ private def readUnderCap(
+ path: String,
+ schema: StructType,
+ storageFilters: Seq[Expression],
+ maxSplicedBytes: String,
+ batchSize: Int,
+ metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
+ readCounting(path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), metrics))
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the long column at `ordinal` of every row.
+ private def readLongs(
+ readerFn: PartitionedFile => Iterator[InternalRow], path: String, ordinal: Int): Seq[Long] = {
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ drain(readerFn(file)) { it =>
+ it.flatMap {
+ case batch: ColumnarBatch => (0 until batch.numRows()).map(batch.column(ordinal).getLong)
+ case row: InternalRow => Seq(row.getLong(ordinal))
+ }.toSeq
+ }
+ }
+
+ // Runs `body`, and returns its result with whether a log line containing `text` was logged.
+ private def withLogged[T](text: String)(body: => T): (T, Boolean) = {
+ val logAppender = new LogAppender(text)
+ var result: Option[T] = None
+ withLogAppender(logAppender) {
+ result = Some(body)
+ }
+ val logged = logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains(text))
+ (result.get, logged)
+ }
+
+ // Runs `body`, and returns its result with whether the reader logged its WARN about reading a
+ // row group without page-level storage filtering.
+ private def withPartialReadWarning[T](body: => T): (T, Boolean) =
+ withLogged("without page-level storage filtering")(body)
+
+ // What the reader logs when filling the survivor buffer fails and it gives splicing up.
+ private val splicingGivenUpOnError = "buffering a surviving key value raised an error"
+
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.fromMap(metricMap())
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // The `(k, v)` schema most fixtures read. `create` reads only the types and names, and the reader
+ // takes each column's repetition from the file, so `nullable` matters only to a format read.
+ private def kvSchema(
+ keyType: DataType = LongType,
+ valueType: DataType = StringType,
+ nullable: Boolean = true): StructType =
+ StructType(Seq(StructField("k", keyType, nullable), StructField("v", valueType, nullable)))
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity, useOffHeap)
+
+ // `k >= threshold` bound to position 0, over a long key.
+ private def keyAtLeast(threshold: Long): Expression =
+ GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = keyAtLeast(threshold)
+ val requested = kvSchema()
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value == 40,
+ s"expected its 40 rows excluded; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping the row group avoids exactly the v chunk a
+ // plain read of this projection reads, and phase 1 still pays for k.
+ val vChunkBytes = footerOf(path).getBlocks.get(0).getColumns.get(1).getTotalSize
+ assert(m.bytesAvoidedByRowGroup.value == vChunkBytes,
+ s"expected the v chunk's $vChunkBytes bytes avoided; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"mixed, $mode vectors: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter, useOffHeap = useOffHeap)
+ try {
+ // Output is exact. Phase 2's column readers are handed `finalRanges` and skip the rows
+ // outside them within a partly kept page, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L)
+ assert(result == expected,
+ s"expected exact filtering; got ${result.map(_._1)}, expected ${expected.map(_._1)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted
+ // has to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ // A mixed projection has non-key bytes to avoid, in both the skipped row groups and
+ // the partially kept one. Asserting they are non-negative would prove nothing, since
+ // `SQLMetric.add` drops a negative and the value cannot go below zero.
+ assert(m.bytesAvoidedByRowGroup.value + m.bytesAvoidedByPageFiltering.value > 0,
+ "a mixed projection with skipped row groups should avoid some non-key bytes")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = keyAtLeast(195L)
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs `initializeInternal`, and it used to run
+ // it a second time on top.
+ withTempDir { dir =>
+ val path = writeKeyParquetFile(dir, 1L to 20L)
+ val requested = kvSchema()
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ var initializations = 0
+ val reader = new VectorizedParquetRecordReader(false, 4096) {
+ override protected def initializeInternal(): Unit = {
+ initializations += 1
+ super.initializeInternal()
+ }
+ }
+ Utils.tryWithResource(reader) { reader =>
+ reader.initialize(split, context)
+ assert(initializations == 1, s"initialized $initializations times")
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = kvSchema()
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(keyAtLeast(0L)),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ bloomLiteralOf(42L),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // A runtime bloom's `Literal` holding one long key, built the way the planner builds one. It is
+ // serialized as `BloomFilterAggregate` serializes it, and the key is hashed with the same seed.
+ private def bloomLiteralOf(key: Long): Literal = {
+ val bf = BloomFilter.create(10, 128)
+ bf.putLong(new XxHash64(Seq(Literal(key))).eval(InternalRow.empty).asInstanceOf[Long])
+ Literal(BloomFilterAggregate.serialize(bf), BinaryType)
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = kvSchema(decimalType)
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val key = OnKey(BoundReference(0, LongType, nullable = false), at = 7L,
+ error = new SparkException(_))
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a task kill or a fatal error behind the key expression's error is not absorbed") {
+ // The fail-open must not swallow what the executor has to see. A kill interrupts the task's
+ // thread, and a UDF that wraps the interrupt raises its own error, so the kill is read off the
+ // task context rather than off the error. It goes out wrapped in a checked exception, since
+ // `FileScanRDD` reads a RuntimeException from a reader as a corrupt file under
+ // `ignoreCorruptFiles` and would skip the rest of it. A fatal error an expression wrapped is
+ // found in the cause chain, which is where the executor looks for one, and goes out as itself.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = kvSchema()
+ val bound = BoundReference(0, LongType, nullable = false)
+ def readWith(key: Expression): SparkException = intercept[SparkException] {
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ readAll(path, filter)._2.close()
+ }
+ def assertPropagates(thrown: SparkException, original: Class[_]): Unit = {
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"ignoreCorruptFiles must not be able to read this as a corrupt file; got $thrown")
+ assert(original.isInstance(thrown.getCause) &&
+ thrown.getCause.getMessage.contains("refuses 7"),
+ s"the key expression's own error must reach the task behind it; got ${thrown.getCause}")
+ }
+
+ def killedWhile(key: Expression): SparkException = {
+ TaskContext.setTaskContext(TaskContext.empty())
+ try readWith(key) finally TaskContext.unset()
+ }
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new SparkException(_))),
+ classOf[SparkException])
+ assertPropagates(
+ killedWhile(OnKey(bound, at = 7L, killsTask = true, error = new IllegalStateException(_))),
+ classOf[IllegalStateException])
+
+ // An unchecked error around an OutOfMemoryError, the way a UDF's catch-all wraps one.
+ val fatal = intercept[OutOfMemoryError] {
+ val key = OnKey(bound, at = 7L,
+ error = new IllegalStateException(_, new OutOfMemoryError("wrapped")))
+ readAll(path, createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested))._2
+ .close()
+ }
+ assert(fatal.getMessage == "wrapped", s"the wrapped error itself must go out; got $fatal")
+
+ // An `InternalError` goes out in a checked wrapper, since `ignoreCorruptFiles` would take
+ // the bare one for a corrupt file, where a plain plan raises it from the post-scan Filter.
+ assertPropagates(readWith(OnKey(bound, at = 7L, error = new InternalError(_))),
+ classOf[InternalError])
+
+ // An AssertionError is not fatal, so the reader falls back on it as on any other error.
+ val assertingKey = OnKey(bound, at = 7L, error = new AssertionError(_))
+ val (result, reader) =
+ readAll(path, createFilter(Seq(GreaterThanOrEqual(assertingKey, Literal(10L))), requested))
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a fatal error goes out as itself, found as deep as the executor would find it") {
+ // A plain plan raises a filter's error from the post-scan Filter as it is, and a decoding error
+ // from the reader, which `FileScanRDD` wraps in FAILED_READ_FILE. The executor looks
+ // `spark.executor.killOnFatalError.depth` levels down from what reaches it, so a decoding
+ // error's fatal cause counts down to one level less. An `InternalError` from decoding is
+ // corrupt data, which the plain read meets again.
+ val depth = SparkEnv.get.conf.get(KILL_ON_FATAL_ERROR_DEPTH)
+ def under(levels: Int, fatal: Throwable): Throwable =
+ (0 until levels).foldLeft(fatal)((cause, _) => new IllegalStateException("wraps", cause))
+ val oom = new OutOfMemoryError("fatal")
+
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagate(under(depth - 1, oom))
+ } eq oom, "the deepest level the executor sees counts")
+ ParquetStorageFilter.rethrowIfMustPropagate(under(depth, oom))
+
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(depth - 2, oom))
+ } eq oom, "a decoding error's chain counts one level less deep")
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(depth - 1, oom))
+
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(under(1, new InternalError("codec")))
+ // One from an expression goes out in a checked wrapper, raw or wrapped. `ignoreCorruptFiles`
+ // would take the bare `InternalError` for a corrupt file, and the executor finds it under it.
+ Seq(new InternalError("expression"), under(1, new InternalError("expression"))).foreach { e =>
+ val thrown = intercept[SparkException](ParquetStorageFilter.rethrowIfMustPropagate(e))
+ assert(thrown.getCause.isInstanceOf[InternalError], s"got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got $thrown")
+ assert(Executor.isFatalError(thrown, depth), s"got $thrown")
+ }
+ ParquetStorageFilter.rethrowIfMustPropagate(new AssertionError("not fatal"))
+
+ // A vector only the reader allocates that cannot grow, the way `reserve` reports it, is the
+ // reader's own extra work while decoding. Raised from an expression it is a wrapped OOM.
+ val growth = new RuntimeException("Cannot reserve", new OutOfMemoryError("reserve"))
+ ParquetStorageFilter.rethrowIfMustPropagateFromRead(growth)
+ assert(intercept[OutOfMemoryError] {
+ ParquetStorageFilter.rethrowIfMustPropagate(growth)
+ }.getMessage == "reserve")
+
+ // A fatal error at the top goes out as it is even with a kill pending, as the executor checks
+ // it first. One deeper in the chain goes out as the kill, which the executor reports.
+ val context = TaskContext.empty()
+ context.markInterrupted("killed by the test")
+ TaskContext.setTaskContext(context)
+ try {
+ assert(intercept[OutOfMemoryError](ParquetStorageFilter.rethrowIfMustPropagate(oom)) eq oom)
+ val killed = intercept[SparkException] {
+ ParquetStorageFilter.rethrowIfMustPropagate(under(1, oom))
+ }
+ assert(killed.getCause.isInstanceOf[IllegalStateException], s"got $killed")
+ } finally {
+ TaskContext.unset()
+ }
+ }
+
+ test("a task kill is seen while the reader works through a row group") {
+ // A plain read meets a kill in `FileScanRDD` after every batch. This reader decodes and
+ // evaluates a whole row group before its first batch, and can pass row groups the filter
+ // empties without emitting one, so it checks for a kill itself, per chunk and per row group.
+ withTempDir { dir =>
+ val rows = (0L until 100L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val bound = BoundReference(0, LongType, nullable = false)
+ // Marks the task killed at row 5, inside phase 1's first chunk of 16 rows. With one row
+ // group, only the check at the next chunk can see it before the read ends.
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(OnKey(bound, at = 5L, killsTask = true), Literal(50L))), kvSchema())
+ TaskContext.setTaskContext(TaskContext.empty())
+ val thrown = try {
+ intercept[SparkException] {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16)._2.close()
+ }
+ } finally {
+ TaskContext.unset()
+ }
+ assert(thrown.getCause.isInstanceOf[TaskKilledException],
+ s"the kill must reach the task; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
+ s"and ignoreCorruptFiles must not be able to swallow it; got $thrown")
+ }
+ }
+
+ // Returns the FileSourceScanExec `df` plans to, with the storage filters `build` makes attached.
+ // `build` resolves a column by name against the scan's own output. Goes through Spark's normal
+ // planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def withStorageFilters(df: DataFrame)(
+ build: (String => Attribute) => Seq[Expression]): FileSourceScanExec = {
+ val scan = scanOf(df.queryExecution.executedPlan)
+ def attr(name: String): Attribute =
+ scan.output.find(_.name == name).getOrElse(fail(s"No $name in scan output"))
+ scan.copy(storageFilters = build(attr))
+ }
+
+ // The scan of `df` with a `k >= threshold` storage filter attached.
+ private def scanWithStorageFilter(df: DataFrame, threshold: Long): FileSourceScanExec =
+ withStorageFilters(df)(attr => Seq(GreaterThanOrEqual(attr("k"), Literal(threshold))))
+
+ // The same over `SELECT k, v` of the file at `path`.
+ private def scanWithStorageFilter(path: String, threshold: Long): FileSourceScanExec =
+ scanWithStorageFilter(spark.read.parquet(path).select("k", "v"), threshold)
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def collectRows(plan: SparkPlan): Array[InternalRow] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect()
+ }
+
+ // `collectRows` over a `(k, v)` output.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] =
+ collectRows(plan).map(r => (r.getLong(0), r.getString(1)))
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, threshold = 195L)
+ val collected = executePlanCollect(scan).toSeq.sorted
+ val expected = rows.filter(_._1 >= 195L)
+ assert(collected == expected, s"got $collected; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. The pushed filter drops pages, phase 1 reads the key over the pages it kept and
+ // narrows them with the storage filter, and phase 2 reads the non-key column under the
+ // intersection. Each filter alone keeps rows the other drops, so the scan returns exactly the
+ // rows both keep.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ // Pages hold 100 rows, so this drops the first page and keeps the other three.
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_0100'")
+ // Keeps the first page, which the pushed filter drops, and drops the last 150 rows.
+ val withSF =
+ withStorageFilters(df)(attr => Seq(LessThanOrEqual(attr("k"), Literal(250L))))
+ val pushedOnly = executePlanCollect(withSF.copy(storageFilters = Nil)).toSeq.sorted
+ assert(pushedOnly == rows.drop(100),
+ s"the pushed filter must drop the first page; got ${pushedOnly.size} rows")
+
+ val collected = executePlanCollect(withSF).toSeq.sorted
+ assert(collected == rows.slice(100, 250),
+ s"expected keys 101..250, the rows both filters keep; got ${collected.size} rows")
+ val excludedWithin =
+ withSF.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excludedWithin == 150,
+ s"the storage filter excludes 150 of the 300 rows the pushed filter kept; got " +
+ excludedWithin)
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ Seq(false, true).foreach { ignoreCorruptFiles =>
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too " +
+ s"(ignoreCorruptFiles = $ignoreCorruptFiles)") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it. ignoreCorruptFiles does not turn
+ // the feature off.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> ignoreCorruptFiles.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSeq == Seq((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since
+ // the post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty)
+ .getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSeq == Seq(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("a row group over the splice cap reads its key columns again: same rows, more bytes, " +
+ "less avoided") {
+ // Past the cap a row group gives splicing up. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Three things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. The
+ // fallback must actually have happened, which the second read of the key column shows as
+ // strictly more bytes transferred. And that second read is a cost against the saving, so the
+ // byte metric must report less avoided than while splicing.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group that gave splicing up still has to emit
+ // in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val path = writePagedFile(dir)
+
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ val metrics = metricMap()
+ val (rows, bytes) = readUnderCap(path, kvSchema(), Seq(keyAtLeast(threshold)),
+ maxSplicedBytes, batchSize = 16, metrics)
+ (rows, bytes, metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ }
+
+ val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ // `k >= 350` keeps one contiguous range of 40 bytes, and a long key buffers 9 bytes per
+ // survivor, so a 100-byte cap is passed at the seventh.
+ val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ assert(splicedAvoided > 0, s"page filtering has to avoid something here; got $splicedAvoided")
+ assert(plainAvoided < splicedAvoided,
+ s"the second key read must count against the saving; plain=$plainAvoided " +
+ s"spliced=$splicedAvoided")
+
+ // The two halves of the budget are weighed together, and this is the case that says so.
+ // Eight survivors buffer 72 bytes and fall in one range of 40, so neither half reaches the
+ // 100-byte cap on its own while the sum passes it at the seventh. Weighing them separately
+ // would keep splicing here, and report the larger saving for it.
+ val (_, _, eightSpliced) = run("64MB", 393L)
+ val (_, _, eightPlain) = run("100b", 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+
+ // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 bytes, which a cap of
+ // 103 holds and one of 102 does not. A charge one byte off either way flips one of the two.
+ val (_, sevenUncapped, _) = run("64MB", 394L)
+ val (_, sevenAtCharge, _) = run("103b", 394L)
+ val (_, sevenUnderCharge, _) = run("102b", 394L)
+ assert(sevenAtCharge == sevenUncapped,
+ s"a cap of exactly the charge must keep splicing; got $sevenAtCharge against " +
+ sevenUncapped)
+ assert(sevenUnderCharge > sevenUncapped,
+ s"a cap one byte under it must give splicing up; got $sevenUnderCharge against " +
+ sevenUncapped)
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups. Each row
+ // group's path is asserted too, as whether its batches hold the persistent key vector, which
+ // also pins the charge: at twice the long key's 9 bytes the first row group would pass the cap.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row groups")
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val reader = new LateMaterializationParquetRecordReader(false, 16, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ // The batch holds the persistent vectors until the first batch splices.
+ val persistentKeys = reader.resultBatch().column(0)
+ val result = mutable.ArrayBuffer[(Long, String)]()
+ val splicedByRowGroup = mutable.LinkedHashMap[Long, Set[Boolean]]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ val spliced = batch.column(0) ne persistentKeys
+ (0 until batch.numRows()).foreach { i =>
+ val key = batch.column(0).getLong(i)
+ result += ((key, batch.column(1).getUTF8String(i).toString))
+ val rowGroup = (key - 1) / 100
+ splicedByRowGroup(rowGroup) = splicedByRowGroup.getOrElse(rowGroup, Set()) + spliced
+ }
+ }
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(splicedByRowGroup == Map(0L -> Set(true), 1L -> Set(false), 2L -> Set(true)),
+ s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got $splicedByRowGroup")
+ assert(m.rowGroupsSkipped.value == 1,
+ s"expected the row group with no survivor skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a later accumulator set is sized up front only when its charge fits under the cap") {
+ // Every set after the first sizes a variable-length key's byte child to what the set before it
+ // held, which takes that memory before any value arrives, so it is charged then. Six survivors
+ // of a 1000-byte key over a capacity of 4 fill one set of four, charged about 16KB, and sizing
+ // the next set would charge as much again for two survivors. Under a cap of 28000 bytes that
+ // does not fit, so the set grows value by value instead, and the row group stays spliced, which
+ // reading the key column only once shows. The sizing has to fit after the fifth survivor's own
+ // fixed 17 bytes too. A cap of 32120 leaves 15995 for it, so the set grows and stays spliced.
+ // Under a cap that fits it, the second set is sized and charged once. A cap of 32130 holds
+ // the 32125 bytes of the fifth survivor and gives splicing up at the sixth, at 32142. A cap
+ // of 40000 keeps splicing. A missing or a double charge would each get one of these wrong.
+ // And the keys that land in the second set must come back right either way.
+ withTempDir { dir =>
+ val df = spark.range(1, 21).selectExpr(
+ "CONCAT(LPAD(CAST(id AS STRING), 4, '0'), REPEAT('x', 996)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row group")
+ val fileSchema = kvSchema(keyType = StringType)
+ val storageFilters = Seq(GreaterThanOrEqual(
+ BoundReference(0, StringType, nullable = true), Literal.create("0015", StringType)))
+
+ def run(maxSplicedBytes: String): (Int, Long) =
+ readUnderCap(path, fileSchema, storageFilters, maxSplicedBytes, batchSize = 4)
+
+ val (uncappedRows, uncappedBytes) = run("64MB")
+ val (cappedRows, cappedBytes) = run("28000b")
+ assert(uncappedRows == 6 && cappedRows == 6,
+ s"the filter keeps keys 15..20 either way; got $uncappedRows and $cappedRows")
+ assert(cappedBytes == uncappedBytes,
+ "a set whose sizing does not fit must grow instead, so the row group stays spliced and " +
+ s"reads the key column once; capped=$cappedBytes uncapped=$uncappedBytes")
+ val (_, noRoomBytes) = run("32120b")
+ assert(noRoomBytes == uncappedBytes,
+ "a sizing that fits only without this survivor's own charge must not happen, so a cap of " +
+ s"32120 bytes keeps splicing; got $noRoomBytes against $uncappedBytes")
+ val (_, justUnderBytes) = run("32130b")
+ assert(justUnderBytes > uncappedBytes,
+ "the sizing must be charged when the set is sized, which passes a cap of 32130 bytes; " +
+ s"got $justUnderBytes against $uncappedBytes")
+ val (_, roomyBytes) = run("40000b")
+ assert(roomyBytes == uncappedBytes,
+ "and charged once, so a cap of 40000 bytes keeps splicing; " +
+ s"got $roomyBytes against $uncappedBytes")
+
+ val expected = (15 to 20).map(id => (f"$id%04d" + "x" * 996, id.toString))
+ Seq(Long.MaxValue, 28000L).foreach { cap =>
+ val (pairs, reader) = readAllWith(
+ path, Seq("k", "v"), createFilter(storageFilters, fileSchema, allMetrics(), cap),
+ (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getUTF8String(i).toString),
+ capacity = 4)
+ try {
+ assert(pairs == expected,
+ s"every survivor must come back with its own key under cap $cap; got " +
+ pairs.map(_._2))
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The footer check sees the missing index
+ // for every row group before anything is read, so no read fails here. The middle row group is
+ // emptied by the filter and skipped whole, and the other two keep rows and are read whole.
+ // Spark's own writer always writes the page index, hence the hand-built file. The retry after a
+ // failing read is covered by the test on a file whose index parquet cannot parse.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, _) => false)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = kvSchema(valueType = LongType)
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ val ((result, reader), warned) = withPartialReadWarning {
+ readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!warned, "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index is declined by the footer check, not by the read") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, which
+ // wastes the buffer and reads the row group again whole. The rows would not show that, since
+ // the row group comes back whole either way. The WARN the exception path logs is what does.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters = Seq(keyAtLeast(50L))
+ val (values, warned) = withPartialReadWarning {
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 1)
+ }
+ // The footer check gets there before the phase-2 read does, which is what keeps the rest of
+ // the file's page filtering. This WARN is what the exception path would have reported.
+ assert(!warned,
+ "the footer check must decline the row group rather than the read discovering it")
+ // The row group that cannot be narrowed comes back whole, since its filter is given up and
+ // the empty store widens its ranges to the block.
+ assert((1L to 100L).forall(values.contains),
+ s"the first row group must come back whole; got ${values.count(_ <= 100L)} of its rows")
+ // The other two keep their page pruning. Read whole they would contribute 200 rows rather
+ // than one page each.
+ assert(values.size < 300,
+ s"the row groups that can be narrowed must still be; got all ${values.size} rows")
+ // And every row the pushed filter actually matches survives the narrowing.
+ assert(Seq(1L, 101L, 201L).forall(values.contains),
+ s"the matching rows must be in the output; got ${values.size} rows")
+ }
+ }
+
+ // Overwrites the bytes of one chunk's offset index while leaving the footer's reference to it.
+ // Parquet then reads the reference, fails to parse what it points at, and reports the whole block
+ // as having no column index, which is the one file shape that reaches phase 2's retry.
+ private def breakOffsetIndex(filePath: String, blockIdx: Int, columnIdx: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(blockIdx).getColumns.get(columnIdx)
+ val ref = chunk.getOffsetIndexReference
+ assert(ref != null, s"block $blockIdx column $columnIdx has no offset index to damage")
+ overwriteBytes(filePath, ref.getOffset, ref.getLength)
+ }
+
+ // Overwrites the back half of one data page of `column` in the first row group, which leaves the
+ // page header, and so everything parquet reads eagerly, intact and the page body undecodable.
+ private def corruptDataPage(filePath: String, column: String, pageIdx: Int): Unit = {
+ val (offset, size) = Utils.tryWithResource(ParquetFileReader.open(
+ HadoopInputFile.fromPath(new Path(filePath), spark.sessionState.newHadoopConf()))) {
+ reader =>
+ val chunk = reader.getRowGroups.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ val index = reader.readOffsetIndex(chunk)
+ (index.getOffset(pageIdx), index.getCompressedPageSize(pageIdx))
+ }
+ overwriteBytes(filePath, offset + size / 2, size - size / 2)
+ }
+
+ // Rewrites the dictionary page header of `column` in the first row group to hold only `entries`
+ // values, so the ids past them point at nothing. The page itself is untouched, and a column that
+ // decodes lazily meets the damage only when one of those values is read.
+ private def shrinkDictionary(filePath: String, column: String, entries: Int): Unit = {
+ val chunk = footerOf(filePath).getBlocks.get(0).getColumns.asScala
+ .find(_.getPath.toDotString == column).getOrElse(fail(s"no $column chunk"))
+ assert(chunk.hasDictionaryPage, s"$column has no dictionary page to damage")
+ val bytes = new Array[Byte](256)
+ val raf = new RandomAccessFile(filePath, "r")
+ try {
+ raf.seek(chunk.getDictionaryPageOffset)
+ raf.readFully(bytes)
+ } finally {
+ raf.close()
+ }
+ val in = new ByteArrayInputStream(bytes)
+ val header = Util.readPageHeader(in)
+ val headerLength = bytes.length - in.available()
+ assert(header.getDictionary_page_header.getNum_values > entries,
+ s"$column's dictionary must hold more than $entries values")
+ header.getDictionary_page_header.setNum_values(entries)
+ val rewritten = new ByteArrayOutputStream()
+ Util.writePageHeader(header, rewritten)
+ assert(rewritten.size() == headerLength, "the rewritten header must keep its length")
+ writeBytes(filePath, chunk.getDictionaryPageOffset, rewritten.toByteArray)
+ }
+
+ // Overwrites `length` bytes at `offset` with 0xFF.
+ private def overwriteBytes(filePath: String, offset: Long, length: Int): Unit =
+ writeBytes(filePath, offset, Array.fill(length)(0xFF.toByte))
+
+ // Writes `bytes` over the file at `offset`.
+ private def writeBytes(filePath: String, offset: Long, bytes: Array[Byte]): Unit = {
+ val file = new File(filePath)
+ val raf = new RandomAccessFile(file, "rw")
+ try {
+ raf.seek(offset)
+ raf.write(bytes)
+ } finally {
+ raf.close()
+ }
+ // Hadoop's local filesystem checks a side-car checksum that the write above invalidates, and a
+ // mismatch there would fail the read for the wrong reason.
+ val crc = new File(file.getParentFile, s".${file.getName}.crc")
+ if (crc.exists()) assert(crc.delete(), s"could not remove $crc")
+ }
+
+ test("a corrupt key page gives the row group up and fails where a plain read fails") {
+ // Parquet reads a chunk's bytes up front but decodes a page only when it is read, so a corrupt
+ // key page shows in phase 1. When its decoding raises an exception, as it does here, giving the
+ // row group up hands it to phase 2 whole, which reads the same pages in the same batches as a
+ // plain read, so the page fails at the same batch, after the same rows. That is what makes
+ // `ignoreCorruptFiles` keep exactly what it keeps of a plain read, and it is asserted here on
+ // the readers themselves, as the rows each hands back before it throws. The filter is
+ // selective, so a reader that kept applying it would return fewer. A codec that reports
+ // corruption as an `InternalError`, as hadoop-lzo does, takes the same path.
+ withTempDir { dir =>
+ val rows = (0L until 1000L).map(i => (i, s"v_$i"))
+ val outDir = new File(dir, "damaged").getAbsolutePath
+ // One row group of 100-row pages, so the fifth key page covers rows 400 to 499.
+ rows.toDF("k", "v").repartition(1).write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 64L * 1024 * 1024)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .option(ParquetOutputFormat.PAGE_ROW_COUNT_LIMIT, 100L)
+ .parquet(outDir)
+ val path = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ .head.getAbsolutePath
+ corruptDataPage(path, column = "k", pageIdx = 4)
+
+ def readUntilFailure(storageFilter: ParquetStorageFilter): (Seq[Long], Throwable) = {
+ val reader = if (storageFilter == null) {
+ new VectorizedParquetRecordReader(false, 16)
+ } else {
+ new LateMaterializationParquetRecordReader(false, 16, storageFilter)
+ }
+ val read = mutable.ArrayBuffer[Long]()
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val error = try {
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ null
+ } catch {
+ case NonFatal(e) => e
+ }
+ (read.toSeq, error)
+ } finally {
+ reader.close()
+ }
+ }
+
+ val (plainRows, plainError) = readUntilFailure(null)
+ assert(plainError != null, "the damaged page must fail a plain read, or this proves nothing")
+ assert(plainRows == (0L until 400L), s"a plain read returns the rows before that page's " +
+ s"batch; got ${plainRows.size}")
+ val (filteredRows, filteredError) = readUntilFailure(keyAtLeastFilter(500L))
+ assert(filteredError != null && filteredError.getClass == plainError.getClass,
+ s"the filtering read must fail the same way; got $filteredError, not $plainError")
+ assert(filteredRows == plainRows,
+ s"and after the same rows; got ${filteredRows.size}, not ${plainRows.size}")
+ }
+ }
+
+ test("a key value the filter does not read and cannot be decoded gives splicing up") {
+ // The filter need not read every key on every row, as `coalesce(a, b)` reads `b` only where
+ // `a` is null. The copy into the survivor buffer reads every key, so it can meet a value no
+ // plain read decodes. A dictionary-encoded column decodes a value only when it is read, so a
+ // dictionary that holds fewer values than its ids point at is such a value. Copying it gives
+ // splicing up for the row group, and phase 2 reads the key columns the way a plain read does.
+ // So the read answers where a plain one does, and the filter still applies.
+ withTempDir { dir =>
+ // Two copies of each `b`, so parquet keeps its dictionary. Cut to 64 values, it leaves `b`
+ // undecodable from row 128 on. `a` is null only before that, so the filter never reads a
+ // damaged `b`.
+ val rows = (0 until 200).map { i =>
+ (if (i < 100 && i % 5 == 0) null else s"a_$i", s"b_${i / 2}", i)
+ }
+ val outDir = new File(dir, "dictionary").getAbsolutePath
+ rows.toDF("a", "b", "v").repartition(1).write
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "true")
+ .parquet(outDir)
+ val path = new File(outDir).listFiles((_, name) => name.endsWith(".parquet")).head
+ .getAbsolutePath
+ assert(encodingsOf(path, "b").exists(_.usesDictionary),
+ s"the fixture's whole point; got ${encodingsOf(path, "b")}")
+ shrinkDictionary(path, column = "b", entries = 64)
+ val columns = Seq("a", "b", "v")
+ intercept[ArrayIndexOutOfBoundsException] {
+ readAllWith(path, columns, null, (b, i) => b.column(1).getUTF8String(i).toString)._2
+ .close()
+ }
+
+ val schema = StructType(Seq(
+ StructField("a", StringType), StructField("b", StringType), StructField("v", IntegerType)))
+ val coalesced = Coalesce(Seq(BoundReference(0, StringType, nullable = true),
+ BoundReference(1, StringType, nullable = true)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(coalesced, Literal.create("a_15", StringType))), schema)
+ val ((result, reader), givenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, columns, filter, (b, i) => b.column(2).getInt(i), capacity = 16)
+ }
+ try {
+ // Keeps rows 150-199 among others, whose `b` is damaged.
+ val expected = rows.filter { case (a, b, _) => Option(a).getOrElse(b) >= "a_15" }.map(_._3)
+ assert(expected.contains(150), "the filter must keep a row whose `b` is damaged")
+ assert(result == expected, s"got ${result.size} rows, expected ${expected.size}")
+ assert(givenUp, "the copy must have failed and given splicing up, or this proves nothing")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group whose offset index parquet cannot parse leaves the rest of the file its " +
+ "page pruning") {
+ // The retry reads the damaged row group whole over the pushed data filter's ranges, and only
+ // that one, since parquet keeps a column index store per block. What it must not give up is
+ // the pushed data filter's own page pruning, which a plain read of the same query keeps, nor
+ // the other row groups' narrowing. Dropping either would read more than it needs to.
+ //
+ // One file shape reaches the retry, a footer that references an offset index parquet cannot
+ // then parse, so the fixture is hand-built and then damaged.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 0)
+ // The fixture's whole point. The footer still offers the index the read cannot use, which is
+ // what gets past the footer check and into the retry.
+ assert(footerOf(path).getBlocks.get(1).getColumns.get(0).getOffsetIndexReference != null,
+ "the damaged index must still be referenced, or the footer check declines the row group")
+ val schema = kvSchema(valueType = LongType, nullable = false)
+ // One value per row group, so parquet's statistics keep every one of them and its column
+ // index narrows each to the page holding that value. All three are even, so all three also
+ // survive the storage filter below.
+ val pushed = Seq(sources.In("v", Array[Any](2L, 102L, 202L)))
+ val evenKeys = Seq(EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = false), Literal(2L)), Literal(0L)))
+ def read(storageFilters: Seq[Expression]): Seq[Long] =
+ readLongs(formatReader(schema, pushed, storageFilters), path, ordinal = 0)
+ val (filtered, warned) = withPartialReadWarning(read(evenKeys))
+ assert(warned, "the read must have discovered the damaged index, or this test proves nothing")
+ // A plain read of the same query is the bar. The filtering one must never hand the post-scan
+ // Filter more rows than that.
+ val plain = read(Nil)
+ assert(filtered.size <= plain.size,
+ s"the filtering read must not read more than a plain one; got ${filtered.size} rows " +
+ s"against ${plain.size}")
+ // The sharp form of the same thing. The last row group's index is intact, so its pages are
+ // still pruned after the retry.
+ assert(filtered.count(_ > 200L) < 100,
+ s"the last row group must keep its pruning; got ${filtered.count(_ > 200L)} of its 100")
+ // The damaged row group comes back whole, which is the cost that is inherent to it.
+ assert((101L to 200L).forall(filtered.contains),
+ s"the damaged row group must come back whole; got " +
+ s"${filtered.count(k => k > 100L && k <= 200L)} of its 100")
+ // And every row the pushed filter matches is in the output, all three being even.
+ assert(Seq(2L, 102L, 202L).forall(filtered.contains),
+ s"the matching rows must survive; got ${filtered.size} rows")
+ // The row groups around the damaged one keep the storage filter's narrowing too, so they
+ // return even keys alone.
+ val undamaged = filtered.filter(k => k <= 100L || k > 200L)
+ assert(undamaged.nonEmpty && undamaged.forall(_ % 2 == 0),
+ s"the other row groups must still be narrowed; got ${undamaged.filter(_ % 2 != 0)}")
+ }
+ }
+
+ test("a v2 file's DELTA_BYTE_ARRAY decimal column is read over scattered survivors") {
+ // Phase 2 skips rows inside a page whenever survivors scatter, where a plain read skips only
+ // when a pushed filter cuts a page. The parquet v2 writer stores a FIXED_LEN_BYTE_ARRAY
+ // decimal without a dictionary as DELTA_BYTE_ARRAY, whose reader could not skip until
+ // SPARK-59828, so this read failed where a plain read of the same file did not.
+ withTempDir { dir =>
+ val outDir = new File(dir, "v2").getAbsolutePath
+ spark.range(0, 1000).selectExpr("id AS k", "CAST(id * 1.25 AS DECIMAL(30, 2)) AS d")
+ .repartition(1)
+ .write
+ .option("parquet.writer.version", "PARQUET_2_0")
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .parquet(outDir)
+ val path = new File(outDir).listFiles((_, name) => name.endsWith(".parquet")).head
+ .getAbsolutePath
+ assert(encodingsOf(path, "d").contains(Encoding.DELTA_BYTE_ARRAY),
+ s"the fixture's whole point; got ${encodingsOf(path, "d")}")
+ val everyThirdRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(3L)), Literal(0L))
+ val schema = kvSchema(valueType = DecimalType(30, 2))
+ val (result, reader) = readAllWith(path, Seq("k", "d"),
+ createFilter(Seq(everyThirdRow), schema),
+ (b, i) => (b.column(0).getLong(i), b.column(1).getDecimal(i, 30, 2).toJavaBigDecimal))
+ try {
+ val expected = (0L until 1000L by 3L)
+ .map(k => (k, new java.math.BigDecimal(k).multiply(new java.math.BigDecimal("1.25"))
+ .setScale(2)))
+ assert(result == expected, s"got ${result.take(5)} (${result.size} rows)")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") {
+ // An alternating filter makes one range per surviving row, which is the shape that exercises
+ // phase 2's range walk and the only one where a defect there is invisible from the row count.
+ // The keys come from the survivor queue and would still be right while the values came from
+ // other rows. So this asserts the pairs.
+ //
+ // Past the cap those ranges cost more than the budget allows, and the filter is given up for
+ // that row group. Every one of its rows is emitted and the post-scan Filter narrows them.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writePagedFile(dir)
+ val fileSchema = kvSchema()
+ val everyOtherRow = EqualTo(
+ Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L))
+
+ def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
+ val filter =
+ createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ val (emitted, reader) = readAll(path, filter, capacity)
+ try emitted finally reader.close()
+ }
+
+ assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ "with room for the ranges, every surviving row comes back with its own value")
+ // Each survivor costs 9 bytes of long key and 40 of its own range. The sum passes the cap of
+ // 1024 at survivor 21, which releases the buffer, and the ranges alone at survivor 26, which
+ // gives the filter up. The capacity decides what the buffer holds when it is released. At
+ // 4096 that is one partly filled set of accumulators, and at 16 a full set in the queue as
+ // well. Both have to end with the queue empty and the filter given up. Keeping one without
+ // the other would leave phase 2 reading non-key columns while emit spliced keys from a queue
+ // holding only the first survivors.
+ assert(readWithCap(1024L) == rows,
+ "past the cap the filter is given up and the row group comes back whole")
+ assert(readWithCap(1024L, capacity = 16) == rows,
+ "and the same when the buffer has been rolled over first")
+ }
+ }
+
+ test("a row group that gives the filter up does not take the next one with it") {
+ // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges
+ // pass the budget must not disarm the filter for the rest of the file. The first row group here
+ // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps
+ // filtering, and the two in between are emptied by the filter and skipped whole.
+ withTempDir { dir =>
+ val rows = rows400
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val k = BoundReference(0, LongType, nullable = true)
+ val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
+ EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
+ val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L)))
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(Or(scatteredInFirst, runInLast)),
+ kvSchema(),
+ m,
+ maxSplicedRowGroupBytes = 1024L)
+ val (result, reader) = readAll(path, filter)
+ try {
+ // The first row group comes back whole, the last only its surviving run.
+ val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L)
+ assert(result == expected,
+ s"expected the given-up row group whole and the last filtered; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 2,
+ s"and the two emptied row groups skipped; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") {
+ // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a
+ // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the
+ // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it
+ // on every row of the ranges the pushed filter left, so it does meet that row.
+ //
+ // It is pushed all the same. When the evaluation throws, the reader gives the filter up for
+ // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where
+ // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of
+ // failing, which is what it does with the feature off.
+ withTable("ansi_app", "ansi_build") {
+ // One file, one row group, with the non-numeric row inside it. A separate file would be
+ // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row.
+ spark.range(600)
+ .selectExpr(
+ "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k",
+ "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind")
+ .repartition(1).write.format("parquet").saveAsTable("ansi_app")
+ spark.range(30).selectExpr("id AS k", "id AS v")
+ .write.format("parquet").saveAsTable("ansi_build")
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " +
+ "WHERE b.v = 5 AND a.kind = 'num'"
+ val df = spark.sql(query)
+ val plan = df.queryExecution.executedPlan
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan")
+ assert(countBloomFiltersInStorageFilters(plan) == 1,
+ s"the cast key is pushed.\nPlan:\n$plan")
+ assert(df.collect().map(_.getString(0)).toSeq == Seq("5"),
+ "and the query must run rather than fail on the non-numeric row")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") {
+ // The reader drops the rows a storage filter rejects and the post-scan Filter evaluates it
+ // again on the rows the reader keeps, so the two evaluations have to agree, and a
+ // non-deterministic conjunct has to stay behind. No producer builds one today, hence the
+ // hand-built plan. `InjectRuntimeFilter`'s blooms hash join keys, which are deterministic.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan
+ val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output"))
+
+ def extractedBlooms(valueExpr: Expression): (Int, Int) = {
+ val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ (countBloomFiltersInStorageFilters(physical),
+ countBloomFiltersInPostScanFilters(physical))
+ }
+
+ // As a control, the same bloom over the key alone is offered to the scan, so the arms
+ // differ in exactly one thing. It stays in the post-scan Filter either way.
+ val (deterministicInScan, deterministicPostScan) =
+ extractedBlooms(new XxHash64(Seq(k)))
+ assert(deterministicInScan == 1 && deterministicPostScan == 1,
+ s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " +
+ s"postScan=$deterministicPostScan")
+
+ // `Rand` contributes no reference, so `k` is still the only key column. The planner's
+ // `deterministic` test on the conjunct is the gate that rejects this one, and the only one
+ // that does, since the format's hook answers about shapes and types alone.
+ val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L))))
+ assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic")
+ val (inScan, postScan) = extractedBlooms(nonDeterministic)
+ assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan")
+ assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers conjuncts under the relation's own column names") {
+ // The analyzer spells a reference the way the query did, so under case-insensitive resolution
+ // a bloom can name `_TMP_METADATA_ROW_INDEX` while the relation, and the schema the reader
+ // requests, call the column `_tmp_metadata_row_index`. The planner normalizes what it offers to
+ // the relation's names, so the format's gate sees the name the reader will see and declines.
+ // Without that the gate passed it, `ParquetStorageFilter.create` rejected it on the driver, and
+ // a query that runs with the conf off failed with it on.
+ withTempDir { dir =>
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 50).selectExpr("id AS k", s"id AS $rowIndexName", "CAST(id AS STRING) AS v"),
+ rowGroupSize = 64 * 1024L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val bloomLit = bloomLiteralOf(42L)
+ val relation = spark.read.parquet(path).select("k", rowIndexName, "v")
+ .queryExecution.optimizedPlan
+ def column(name: String): Attribute =
+ relation.output.find(_.name == name).getOrElse(fail(s"no $name in the relation output"))
+
+ def offered(key: Attribute): Seq[Expression] = {
+ val logical = LogicalFilter(
+ BloomFilterMightContain(bloomLit, new XxHash64(Seq(key))), relation)
+ val physical = FileSourceStrategy(logical).headOption
+ .getOrElse(fail(s"FileSourceStrategy did not plan $logical"))
+ scanOf(physical).storageFilters
+ }
+
+ // As a control, an ordinary key spelled in another case is offered, under its own name.
+ val upperK = offered(column("k").withName("K"))
+ assert(upperK.size == 1, s"a bloom over `K` must reach the scan; got $upperK")
+ assert(upperK.head.references.map(_.name).toSet == Set("k"),
+ s"and it must carry the relation's name for the column; got ${upperK.head}")
+
+ val upperRowIndex =
+ offered(column(rowIndexName).withName(rowIndexName.toUpperCase(Locale.ROOT)))
+ assert(upperRowIndex.isEmpty,
+ s"a bloom over the row-index column must not be offered in any case; got $upperRowIndex")
+ }
+ }
+ }
+
+ // ----- Generic reader plumbing for the coverage tests below -----
+
+ // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need
+ // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`.
+ private def writeKeyParquetFileFromSql(
+ dir: File,
+ keyExpr: String,
+ n: Long = 100L,
+ rowGroupSize: Long = 256L,
+ dictionary: Boolean = false,
+ valueCopies: Int = 1): String = {
+ // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet
+ // actually pick dictionary encoding when it is asked for. A dictionary writer falls back to
+ // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and
+ // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the
+ // decision is per column chunk, and a row group holding one copy of each value is no better
+ // off than a row group of distinct ones.
+ // `div`, not `/`, since `/` is floating-point division in Spark SQL, which would hand every
+ // key expression a double and quietly change the written type.
+ val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id")
+ writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize, dictionary = dictionary)
+ }
+
+ // Every column chunk of the file, from its footer. The facts the tests below assert about a
+ // fixture are read off these rather than assumed from the write options.
+ private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] =
+ footerOf(filePath).getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq
+
+ private def footerOf(filePath: String): ParquetMetadata = {
+ val conf = spark.sessionState.newHadoopConf()
+ ParquetFooterReader.readFooter(
+ HadoopInputFile.fromPath(new Path(filePath), conf), ParquetMetadataConverter.NO_FILTER)
+ }
+
+ // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it.
+ private def encodingsOf(filePath: String, column: String = "k"): Set[Encoding] =
+ footerChunks(filePath).filter(_.getPath.toDotString == column)
+ .flatMap(_.getEncodings.asScala).toSet
+
+ // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which
+ // builds the plain vectorized reader rather than the late-materialization one. Every read helper
+ // in this suite goes through here.
+ //
+ // `tryInitializeResource` closes the reader if anything inside throws and leaves it open
+ // otherwise, which is the contract these helpers need. The caller closes it once its assertions
+ // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks
+ // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade
+ // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at
+ // construction.
+ private def readAllWith[T](
+ filePath: String,
+ columns: Seq[String],
+ storageFilter: ParquetStorageFilter,
+ extract: (ColumnarBatch, Int) => T,
+ capacity: Int = 4096,
+ useOffHeap: Boolean = false,
+ partitionColumns: StructType = new StructType(),
+ partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = {
+ Utils.tryInitializeResource {
+ if (storageFilter == null) {
+ new VectorizedParquetRecordReader(useOffHeap, capacity)
+ } else {
+ new LateMaterializationParquetRecordReader(useOffHeap, capacity, storageFilter)
+ }
+ } { reader =>
+ reader.initialize(filePath, columns.asJava)
+ reader.initBatch(partitionColumns, partitionValues)
+ val collected = mutable.ArrayBuffer[T]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ var i = 0
+ val n = batch.numRows()
+ while (i < n) {
+ collected += extract(batch, i)
+ i += 1
+ }
+ }
+ (collected.toSeq, reader)
+ }
+ }
+
+ // A key value as a string in its *internal* representation, so the splicing path and the plain
+ // path can be compared without knowing the type and without external type conversion.
+ // `InternalRow.getAccessor` does the per-type read, which is why this takes the batch's row
+ // rather than the column vector.
+ private def renderValue(row: InternalRow, ordinal: Int, dt: DataType): String =
+ InternalRow.getAccessor(dt)(row, ordinal) match {
+ case null => "null"
+ // Two equal arrays are two objects, so the default toString would compare identities.
+ case bytes: Array[Byte] => bytes.mkString(",")
+ case value => value.toString
+ }
+
+ // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given
+ // bound predicate accepts. This is the oracle for the splicing path. Whatever the plain reader
+ // returns, filtered in Scala, is exactly what splicing must produce.
+ private def survivorsViaPlainPath(
+ filePath: String,
+ dt: DataType,
+ boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = {
+ val predicate = Predicate.create(boundPredicate)
+ val (rows, reader) = readAllWith(
+ filePath, Seq("k"), null,
+ (b, i) => (renderValue(b.getRow(i), 0, dt), b.getRow(i).copy()))
+ try {
+ rows.collect { case (rendered, row) if predicate.eval(row) => rendered }
+ } finally {
+ reader.close()
+ }
+ }
+
+ // ----- Key-type coverage, one case per ValueCopier branch -----
+
+ // (case name, SQL expression producing `k`, Spark type, threshold as an external value)
+ private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq(
+ ("boolean", "id % 2 = 0", BooleanType, true),
+ ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte),
+ ("short", "CAST(id AS SHORT)", ShortType, 90.toShort),
+ ("int", "CAST(id AS INT)", IntegerType, 90),
+ ("long", "id", LongType, 90L),
+ ("float", "CAST(id AS FLOAT)", FloatType, 90.0f),
+ ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d),
+ ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"),
+ ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)),
+ ("timestamp",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')",
+ TimestampType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)
+ .atZone(java.time.ZoneId.systemDefault()).toInstant),
+ ("timestamp_ntz",
+ "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')",
+ TimestampNTZType,
+ java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)),
+ ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))",
+ YearMonthIntervalType(), java.time.Period.ofMonths(90)),
+ ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))",
+ DayTimeIntervalType(), java.time.Duration.ofSeconds(90)),
+ ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")),
+ ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")),
+ ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")),
+ ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType,
+ "00090".getBytes("UTF-8")))
+
+ // Run every key type both without and WITH dictionary encoding. Dictionary encoding is
+ // parquet's production default, and it is the case where the phase 1 scratch vectors carry a
+ // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's
+ // decode branch rather than straight out of the value array.
+ for {
+ (name, keyExpr, dt, threshold) <- keyTypeCases
+ // Two types have no dictionary arm to take, and parquet's own writer factory says why. There is
+ // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a
+ // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is
+ // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would
+ // be the plain one over again.
+ dictionary <-
+ if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false)
+ else Seq(false, true)
+ } {
+ val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded"
+ test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") {
+ // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with
+ // int96AsTimestamp and which is not the INT64 copier branch we want to cover here.
+ withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") {
+ withTempDir { dir =>
+ // Three copies of every value on the dictionary arm, so the writer keeps the dictionary
+ // instead of falling back to PLAIN.
+ val path = writeKeyParquetFileFromSql(
+ dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1)
+ assert(encodingsOf(path).exists(_.usesDictionary) == dictionary,
+ s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}")
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true), Literal.create(threshold, dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1),
+ s"the $name case should keep some but not all rows; kept ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ // A copier that fails gives splicing up, and phase 2 then reads the keys plainly, so the
+ // rows alone would not show it.
+ val ((result, reader), splicingGivenUp) = withLogged(splicingGivenUpOnError) {
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ }
+ try {
+ assert(result == expected,
+ s"splicing path disagrees with the plain path for $name ($encoding):\n" +
+ s" splicing: $result\n plain: $expected")
+ assert(!splicingGivenUp, s"every $name survivor must be copied into the buffer")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+ }
+
+ test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") {
+ // This is a regression test. TimeType is an AtomicType that passes every planning-time gate
+ // (it is batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a
+ // TIME join key), so a key-type whitelist that omitted it would fail the task at reader init.
+ assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)),
+ "TIME must be an eligible storage-filter key type")
+ withTempDir { dir =>
+ val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong))
+ val path = writeKeyParquetFile(dir, times, rowGroupSize = 256L)
+ val dt = TimeType(6)
+ val bound = GreaterThanOrEqual(
+ BoundReference(0, dt, nullable = true),
+ Literal.create(LocalTime.ofSecondOfDay(90L), dt))
+ val expected = survivorsViaPlainPath(path, dt, bound)
+ assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}")
+
+ val requested = kvSchema(dt)
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, dt))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("isSupportedKeyType covers exactly the types the reader can copy") {
+ // What FileSourceStrategy offers, what ParquetStorageFilter.create admits and what the reader
+ // can copy are one list, because isSupportedKeyType answers from ValueCopier. Both are asserted
+ // here. A type the copier has no case for would be a task failure rather than a planning-time
+ // rejection, and the delegation is what rules that out.
+ val supported: Seq[DataType] = Seq(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
+ DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType,
+ TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(),
+ StringType, VarcharType(10), CharType(10), BinaryType)
+ supported.foreach { dt =>
+ assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type")
+ assert(ValueCopier.forType(dt) != null, s"$dt should have a value copier")
+ }
+ // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types.
+ val unsupported: Seq[DataType] = Seq(
+ VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType),
+ new StructType().add("a", IntegerType))
+ unsupported.foreach { dt =>
+ assert(!ParquetStorageFilter.isSupportedKeyType(dt),
+ s"$dt should NOT be a supported key type")
+ val e = intercept[SparkException](ValueCopier.forType(dt))
+ assert(e.isInternalError && !DataSourceUtils.shouldIgnoreCorruptFileException(e),
+ s"an unsupported key type is the reader's bug, which must fail loudly; got $e")
+ }
+ }
+
+ // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time -----
+
+ test("nullable key column: surviving null keys are copied through as nulls") {
+ // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its
+ // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(
+ dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END")
+ val ref = BoundReference(0, LongType, nullable = true)
+ val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L)))
+ val expected = survivorsViaPlainPath(path, LongType, bound)
+ assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected")
+
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) =
+ readAllWith(path, Seq("k", "v"), filter, (b, i) => renderValue(b.getRow(i), 0, LongType))
+ try {
+ assert(result == expected, s"got $result; expected $expected")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("key ordinals collected out of order still pair with the right batch slots") {
+ // `ParquetStorageFilter.create` sorts the ordinals it collects, which fixes the key row's
+ // layout. The reader pairs each key column's batch slot with its key-row position, so nothing
+ // depends on that order. Every other fixture mentions its keys in column order, and production
+ // does not promise that, since the conjuncts arrive in `afterScanFilters` order, which follows
+ // the join. So this has two key columns mentioned the other way round, and asserts both the
+ // sorted layout and that every key still comes back with its own row.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L)
+
+ // a >= 100 AND b <= 300, written so that `b` (ordinal 1) is collected first.
+ val bound = And(
+ LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)),
+ keyAtLeast(100L))
+ val requested = StructType(Seq(
+ StructField("a", LongType, nullable = true),
+ StructField("b", LongType, nullable = true),
+ StructField("c", StringType, nullable = true)))
+ val filter = createFilter(Seq(bound), requested)
+ assert(filter.keyColumnIndices.toSeq == Seq(0, 1),
+ s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}")
+
+ val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i),
+ b.column(2).getUTF8String(i).toString))
+ try {
+ val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i"))
+ assert(result == expected,
+ s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("partition columns are preserved alongside spliced key columns") {
+ // The emit path rewrites the key slots alone, and a partition slot is not one of them. It sits
+ // past the data columns and must keep the constant vector `initBatch` populated for it.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(195L)
+ val partitionColumns = new StructType().add("p", IntegerType)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)),
+ partitionColumns = partitionColumns,
+ partitionValues = InternalRow(7))
+ try {
+ assert(result.map(_._1) == (195L to 200L),
+ s"expected keys 195..200; got ${result.map(_._1)}")
+ assert(result.forall(_._2 == 7),
+ s"every row should carry partition value 7; got ${result.map(_._2).distinct}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ Seq(false, true).foreach { useOffHeap =>
+ val mode = if (useOffHeap) "off-heap" else "on-heap"
+ test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") {
+ // Off-heap is where the close/free hazards actually bite. OffHeapColumnVector.close() frees
+ // the native buffer, so a read after close is a crash rather than stale data. A second close
+ // is harmless, since close() zeroes the addresses it freed. capacity = 16 over 71 survivors
+ // forces 5 emits, each closing the previous emit's dequeued key vectors. The other 29 rows
+ // are what makes a declining reader fail this.
+ withTempDir { dir =>
+ val path = writeKeyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L)
+ val bound = keyAtLeast(30L)
+ val requested = kvSchema()
+ val filter = createFilter(Seq(bound), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i),
+ capacity = 16, useOffHeap = useOffHeap)
+ try {
+ assert(result == (30L to 100L),
+ s"expected the 71 surviving keys in order; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ }
+
+ test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") {
+ // One ColumnarBatch is handed out for the whole read and its key slots are rewritten per
+ // batch, so a consumer holding on to an earlier getCurrentValue() would read the wrong
+ // vectors. Drives the non-columnar contract with
+ // a capacity small enough to span several batches.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val filter = keyAtLeastFilter(100L)
+ val reader = new LateMaterializationParquetRecordReader(false, 8, filter)
+ try {
+ reader.initialize(path, Seq("k", "v").asJava)
+ reader.initBatch(new StructType(), null)
+ val collected = mutable.ArrayBuffer[(Long, String)]()
+ while (reader.nextKeyValue()) {
+ val row = reader.getCurrentValue().asInstanceOf[InternalRow]
+ collected += ((row.getLong(0), row.getString(1)))
+ }
+ assert(collected.toSeq == rows.filter(_._1 >= 100L),
+ s"expected keys 100..200 row by row; got ${collected.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // ----- Schema evolution, a column missing from the physical file -----
+
+ test("non-key column missing from a file: the evolved table reads and credits avoided bytes") {
+ // Schema evolution end to end, with the non-key column rather than the key missing from the
+ // older file. The clipped requested schema keeps the missing column, which the byte walks skip
+ // by themselves, since they go over the block's chunks and the older file has no chunk for it.
+ // What this pins is that such a read returns the right rows and still credits a saving.
+ withTempDir { dir =>
+ val base = new File(dir, "merged").getAbsolutePath
+ // The older file has (k, v), the newer one (k, v, w).
+ (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+ (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
+ .mode("append")
+ .parquet(base)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w")
+ // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and
+ // drops the leading row groups of the older file so the skip path is exercised there too.
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2))))
+ .toSeq.sorted
+ val expected = (150L to 200L).map(i => (i, s"v_$i", None)) ++
+ (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))
+ assert(collected == expected,
+ s"expected ${expected.size} rows across both schemas; got ${collected.size}")
+
+ // A positive total says the byte walks ran over the older file's row groups as well.
+ // Asserting non-negativity would prove nothing. `SQLMetric.add` drops a negative and the
+ // value cannot go below zero.
+ val bytesRg = withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP)
+ val bytesPf =
+ withSF.metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING)
+ assert(bytesRg.value + bytesPf.value > 0,
+ s"the walk must credit some avoided bytes; got rg=${bytesRg.value} pf=${bytesPf.value}")
+ }
+ }
+ }
+
+ // ----- Explain output -----
+
+ test("StorageFilters shows up in the scan description only when the scan has storage filters") {
+ // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append
+ // `StorageFilters: []` to every file-scan explain line and churn the explain golden files.
+ withTempDir { dir =>
+ val rows = (1L to 20L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val scan = scanOf(spark.read.parquet(path).select("k", "v").queryExecution.executedPlan)
+ assert(!scan.simpleString(100).contains("StorageFilters"),
+ s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}")
+
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L))))
+ assert(withSF.simpleString(100).contains("StorageFilters"),
+ s"a scan with storage filters must mention them: ${withSF.simpleString(100)}")
+ }
+ }
+
+ // ----- Missing KEY column end to end (schema evolution) -----
+
+ // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing
+ // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7".
+ private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause")
+ spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)")
+ body("evolved")
+ }
+ }
+
+ // Attaches `storageFilters` to the scan of `SELECT id, k FROM <table>` and collects the result.
+ private def collectWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): Seq[(Long, Option[Long])] =
+ scanWithStorageFilterOnKey(table, buildFilter)._2
+
+ // As above, and also returns the scan, whose metrics the caller can then read. The rows come
+ // back sorted, so a row returned twice shows.
+ private def scanWithStorageFilterOnKey(
+ table: String,
+ buildFilter: Attribute => Expression): (FileSourceScanExec, Seq[(Long, Option[Long])]) = {
+ val withSF = withStorageFilters(spark.sql(s"SELECT id, k FROM $table")) { attr =>
+ Seq(buildFilter(attr("k")))
+ }
+ // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so
+ // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's.
+ // Resolve positions by name rather than assuming they line up.
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val collected = collectRows(withSF)
+ .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos))))
+ .toSeq.sorted
+ (withSF, collected)
+ }
+
+ test("all keys missing: an error on the constant reads the file rather than skipping it") {
+ // A file with none of the filter's key columns makes the predicate constant over it, so the
+ // reader decides once, from the values `initBatch` materialized for those columns. That
+ // evaluation can throw exactly as a row's value can. Here the older file's `k` reads as its
+ // DEFAULT, which is not a number, and the cast fails under ANSI. Failing open reads the file
+ // the way a plain scan would, so its rows survive. Skipping it would lose them, and rethrowing
+ // would fail a query a plain read answers.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true",
+ SQLConf.ANSI_ENABLED.key -> "true") {
+ withTable("evolved") {
+ spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet")
+ spark.sql("INSERT INTO evolved VALUES (1), (2), (3)")
+ spark.sql("ALTER TABLE evolved ADD COLUMN k STRING DEFAULT 'not-a-number'")
+ spark.sql("INSERT INTO evolved VALUES (4, '40'), (5, '50')")
+ val withSF = withStorageFilters(spark.sql("SELECT id, k FROM evolved")) { attr =>
+ Seq(GreaterThanOrEqual(Cast(attr("k"), LongType), Literal(1L)))
+ }
+ val idPos = withSF.output.indexWhere(_.name == "id")
+ val ids = collectRows(withSF).map(_.getLong(idPos)).toSeq.sorted
+ assert(ids == Seq(1L, 2L, 3L, 4L, 5L),
+ s"the older file must fail open and the newer one pass the cast; got $ids")
+ }
+ }
+ }
+
+ test("all keys missing: a checked exception on the constant reads the file") {
+ // The constant evaluation fails open under the same rule as a row's, a checked exception
+ // included. The older file's `k` reads as its DEFAULT 7, which the key expression throws on,
+ // so that file must be read the way a plain scan would. The newer file's keys evaluate
+ // normally and are rejected, since no Filter runs above this scan.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(OnKey(k, at = 7L, error = new SparkException(_)),
+ Literal(100L)))
+ assert(collected == Seq((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L))),
+ s"the older file must fail open and the newer one be filtered; got $collected")
+ }
+ }
+ }
+
+ test("a file with none of the projected non-key columns is read plainly") {
+ // The clipped requested schema keeps a column the file does not have, so a projected non-key
+ // column added after the older file was written still looks like something for phase 2 to
+ // read. Phase 2 reads nothing for it, so for that file the filter could only add cost, and the
+ // reader declines it there. With no Filter above this scan, the older file's rows come back
+ // unfiltered while the newer file's are still filtered.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTable("evolved_value") {
+ spark.sql("CREATE TABLE evolved_value (k BIGINT, v STRING) USING parquet")
+ spark.sql("INSERT INTO evolved_value VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+ spark.sql("ALTER TABLE evolved_value ADD COLUMN w STRING")
+ spark.sql("INSERT INTO evolved_value VALUES (4, 'd', 'x'), (5, 'e', 'y')")
+ val withSF =
+ scanWithStorageFilter(spark.sql("SELECT k, w FROM evolved_value"), threshold = 5L)
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val keys = collectRows(withSF).map(_.getLong(kPos)).toSeq.sorted
+ assert(keys == Seq(1L, 2L, 3L, 5L),
+ s"the older file must be read unfiltered and the newer one filtered; got $keys")
+ }
+ }
+ }
+
+ test("a file with none of a projected struct's requested fields is read plainly") {
+ // Under the legacy returnNullStructIfAllFieldsMissing, the clipped schema keeps a struct none
+ // of whose requested fields the file has, and adds no field it has, so phase 2 would read
+ // nothing for it. Only the struct's leaves are missing columns, not the struct, so the reader
+ // asks the file schema about every non-key leaf. The filter rejects every row, so a declined
+ // file comes back whole and one that is not comes back empty.
+ withTempDir { dir =>
+ val path = writeSingleParquetFile(dir,
+ spark.range(0, 20).selectExpr("id AS k", "named_struct('a', CAST(id AS INT)) AS s"),
+ rowGroupSize = 64 * 1024L)
+ val schema =
+ new StructType().add("k", LongType).add("s", new StructType().add("b", IntegerType))
+ val storageFilters = Seq(keyAtLeast(1000L))
+ withSQLConf(SQLConf.LEGACY_PARQUET_RETURN_NULL_STRUCT_IF_ALL_FIELDS_MISSING.key -> "true") {
+ val keys = readLongs(formatReader(schema, Nil, storageFilters), path, ordinal = 0)
+ assert(keys == (0L until 20L), s"the file must be read plainly; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT is filtered on the default, not on null") {
+ // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must
+ // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which
+ // is not true, so the whole older file would be skipped and its rows lost.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val collected = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones.
+ val expected: Seq[(Long, Option[Long])] =
+ Seq((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ test("missing key column with an existence DEFAULT that fails the filter skips the older file") {
+ // Mirror image of the previous test. The default does NOT satisfy the predicate, so the older
+ // file must be skipped in full while the newer file is still filtered normally.
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") {
+ withEvolvedKeyTable("k BIGINT DEFAULT 7") { table =>
+ val (scan, collected) = scanWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(30L)))
+ val expected: Seq[(Long, Option[Long])] = Seq((4L, Some(40L)), (5L, Some(50L)))
+ assert(collected == expected, s"got $collected; expected $expected")
+ // Rejecting a file whole is its own metric path, which walks every one of its row groups
+ // rather than going through the per-row-group loop. Nothing else asserts that walk, so it
+ // could report zero and only the rows above would notice.
+ def metric(name: String): Long = scan.metrics(name).value
+ assert(metric(StorageFilterMetrics.ROW_GROUPS_SKIPPED) >= 1,
+ "the older file's row groups count as skipped")
+ assert(metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP) == 3,
+ "and all three of its rows as excluded by a row group, got " +
+ metric(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP))
+ assert(metric(StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP) > 0,
+ "and its projected bytes as avoided")
+ }
+ }
+ }
+
+ test("missing key column with no DEFAULT reads as null and the predicate decides on null") {
+ // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips
+ // the older file and a null-accepting one keeps it. Both directions are checked so the test
+ // pins the semantics rather than just one outcome.
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withEvolvedKeyTable("k BIGINT") { table =>
+ val rejectsNull = collectWithStorageFilterOnKey(
+ table, k => GreaterThanOrEqual(k, Literal(5L)))
+ assert(rejectsNull == Seq((4L, Some(40L)), (5L, Some(50L))),
+ s"a null-rejecting predicate should drop the older file; got $rejectsNull")
+
+ val acceptsNull = collectWithStorageFilterOnKey(
+ table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L))))
+ val expected: Seq[(Long, Option[Long])] =
+ Seq((1L, None), (2L, None), (3L, None), (5L, Some(50L)))
+ assert(acceptsNull == expected,
+ s"a null-accepting predicate should keep the older file; got $acceptsNull")
+ }
+ }
+ }
+
+ // ----- Metadata columns and complex non-key columns -----
+
+ test("_metadata.row_index is correct alongside a spliced key column") {
+ // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2
+ // PageReadStore. It must report absolute row indexes within the file, not positions within the
+ // filtered batch.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select(
+ col("k"), col("v"), col("_metadata.row_index").as("ri"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ val collected = collectRows(withSF).map(r => (r.getLong(0), r.getLong(2))).toSeq.sorted
+ // Rows were written in ascending k order in a single file, so row_index == k - 1.
+ val expected = (150L to 200L).map(k => (k, k - 1))
+ assert(collected == expected,
+ s"row_index must be the absolute index in the file; got ${collected.take(5)}")
+ }
+ }
+ }
+
+ test("complex non-key column is assembled correctly under splicing") {
+ // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a
+ // struct column exercises both, which a flat projection never does.
+ withTempDir { dir =>
+ val outDir = new File(dir, "structs").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s")
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF =
+ scanWithStorageFilter(spark.read.parquet(outDir).select("k", "s"), threshold = 195L)
+ val collected = collectRows(withSF)
+ .map { r =>
+ val s = r.getStruct(1, 2)
+ (r.getLong(0), s.getInt(0), s.getString(1))
+ }.toSeq.sorted
+ val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k"))
+ assert(collected == expected, s"got $collected; expected $expected")
+ }
+ }
+ }
+
+ test("a storage filter binds to the right column with partition and metadata columns in output") {
+ // `preparedStorageFilters` binds the filter's references against `output.take(requiredSchema
+ // .length)`, and `output` is the data columns, then the generated metadata columns, then the
+ // partition columns, then the constant metadata ones. A projection carrying all four is where a
+ // wrong prefix would bind the predicate to another column and drop rows the filter keeps, which
+ // the post-scan Filter cannot put back. The file's first column `a` is not projected, so the
+ // key sits at a different position in the relation's data schema than in the scan's.
+ withTempDir { dir =>
+ val outDir = new File(dir, "parted").getAbsolutePath
+ spark.range(1, 201)
+ .selectExpr("id * 7 AS a", "id AS k", "CONCAT('v_', id) AS v", "CAST(id % 3 AS INT) AS p")
+ .repartition(1)
+ .write
+ .partitionBy("p")
+ .option(ParquetOutputFormat.BLOCK_SIZE, 256L)
+ .parquet(outDir)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(outDir).select(
+ col("k"), col("v"), col("p"),
+ col("_metadata.row_index").as("ri"), col("_metadata.file_size").as("fs"))
+ val withSF = scanWithStorageFilter(df, threshold = 150L)
+ // The assumption the binding rests on, stated where a change to `output` would break it.
+ assert(withSF.output.take(withSF.requiredSchema.length).map(_.name) ==
+ withSF.requiredSchema.fieldNames.toSeq,
+ s"the scan's leading attributes must be its data columns; got " +
+ s"${withSF.output.map(_.name)}")
+ val kPos = withSF.output.indexWhere(_.name == "k")
+ val pPos = withSF.output.indexWhere(_.name == "p")
+ def collectAll(plan: SparkPlan): Seq[Seq[Any]] =
+ collectRows(plan).map(_.toSeq(plan.output.map(_.dataType))).toSeq
+ .sortBy(_(kPos).asInstanceOf[Long])
+ val filtered = collectAll(withSF)
+ // The same scan without the filter is the oracle, so the row indexes and file sizes come
+ // from the read rather than from arithmetic over how the partitions were written.
+ val expected = collectAll(withSF.copy(storageFilters = Nil))
+ .filter(_(kPos).asInstanceOf[Long] >= 150L)
+ assert(filtered == expected,
+ s"expected ${expected.size} rows with their own partition and metadata values; " +
+ s"got ${filtered.size}")
+ assert(filtered.map(_(pPos)).distinct.size == 3,
+ s"every partition value must still be represented; got ${filtered.map(_(pPos)).distinct}")
+ }
+ }
+ }
+
+ // ----- Planner gates -----
+
+ test("supportsStorageFilter is what decides, and a subclass answers false") {
+ // The planner asks the format rather than testing its class, so the expression shapes and the
+ // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass
+ // still answers false. It may customize reading by overriding buildReaderWithPartitionValues,
+ // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which
+ // would bypass whatever the subclass does.
+ //
+ // This hook answers about shapes and types alone. Whether the conjunct is deterministic and
+ // whether its references are projected columns of this scan are plan-visible, so they stay with
+ // the planner; `FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter`
+ // covers that half.
+ val format = new ParquetFileFormat()
+ val subclass = new ParquetFileFormat() {}
+ val bloom = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("k", LongType)()), 42L))
+ val onVariant = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(AttributeReference("v", VariantType)()), 42L))
+ // A cast key is supported. In ANSI mode it can throw on a row an earlier conjunct would have
+ // dropped, and the reader handles that where it arises, by giving the filter up for the row
+ // group. Declining here instead would also decline every widening cast, which is what type
+ // coercion inserts for a join between an int and a bigint column.
+ val onCast = BloomFilterMightContain(
+ Literal.create(null, BinaryType),
+ XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L))
+
+ // The conf is read in the format rather than in the planner, and it is the per-scan question
+ // that carries it, asked once before anything per conjunct.
+ assert(!format.supportsStorageFilterPushdown(spark), "the feature is off with the conf off")
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ assert(format.supportsStorageFilterPushdown(spark), "and on with it on")
+ assert(!subclass.supportsStorageFilterPushdown(spark), "a subclass must not claim support")
+ assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported")
+ // Not a bloom at all, and a bloom on a type the value copier has no branch for.
+ assert(!format.supportsStorageFilter(Literal.TrueLiteral))
+ assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf")
+ assert(format.supportsStorageFilter(onCast),
+ "a cast key is pushed, and an evaluation error gives the row group up")
+ // And the default is no support at all.
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom))
+ }
+ }
+
+
+ test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") {
+ // The rows are right either way, since the bloom stays in the post-scan Filter. What this gate
+ // saves is offering the scan a filter its reader cannot apply.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ assert(countBloomFiltersInStorageFilters(plan) == 0,
+ s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan")
+ }
+ }
+ }
+
+ test("a scan reads plainly when the vectorized reader is disabled after planning") {
+ // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor
+ // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather
+ // than a wrong one. Every row of the file comes back from the scan.
+ withTempDir { dir =>
+ val rows = (1L to 50L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows)
+ val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ scanWithStorageFilter(path, threshold = 25L)
+ }
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ val keys = executePlanCollect(withSF).map(_._1).toSeq.sorted
+ assert(keys == (1L to 50L),
+ s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows")
+ }
+ }
+ }
+
+ test("a file format without storage-filter support declines rather than delegating") {
+ // The default `FileFormat.buildReaderWithStorageFilters` answers None, and the caller falls
+ // back to the ordinary builder. That is what keeps the two builders from being able to call
+ // each other, and declining is safe because the post-scan Filter still holds the conjunct.
+ val storageFilters = Seq(keyAtLeast(1L))
+ val declined = new NoStorageFilterFileFormat().buildReaderWithStorageFilters(
+ spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters,
+ Map.empty, new Configuration(), Map.empty)
+ assert(declined.isEmpty, "the default must not build a reader of its own")
+ assert(!new NoStorageFilterFileFormat().supportsStorageFilterPushdown(spark),
+ "and it must not claim support either")
+ }
+
+ test("a scan whose format declines falls back to the ordinary reader and returns every row") {
+ // The check above is that the default answers None. This is the other half, that `inputRDD`
+ // then builds the ordinary reader and the scan still produces its rows. A `ParquetFileFormat`
+ // subclass is the shape that reaches it, since it inherits the override and declines by class,
+ // and its ordinary builder is the one an extension would have customized.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val withSF = scanWithStorageFilter(path, threshold = 195L)
+ val declining = withSF.copy(
+ relation = withSF.relation.copy(fileFormat = new ParquetFileFormat() {})(spark))
+ val collected = executePlanCollect(declining).toSeq.sorted
+ assert(collected == rows,
+ s"a declining format reads the file plainly; got ${collected.size} of ${rows.size} rows")
+ }
+ }
+ }
+
+ Seq(false, true).foreach { aqe =>
+ test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") {
+ // AQE is on by default in production, and it is where the bloom subquery is planned by
+ // PlanAdaptiveSubqueries rather than PlanSubqueries, which is the path
+ // preparedStorageFilters' ScalarSubquery materialization depends on.
+ withBloomFilterTables {
+ val baseConf = Map(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString)
+ def run(pushdown: Boolean): (Int, Seq[(Long, Long)], Long) = withSQLConf(
+ (baseConf +
+ (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _*
+ ) {
+ val (plan, rows) = runBloomFilterJoin()
+ val excluded = collect(plan) {
Review Comment:
Nit: this sum of the two row-exclusion metrics is also written at L861-866 and L2630-2634, and those two walk the plan with `plan.collect`, which the comment at L791-792 says would report zero for a plan AQE wrapped. They are right today only because their tests turn AQE off. Could one helper next to `countBloomFiltersInStorageFilters` (L801) serve all three, with the `scans` at L2630 taken through `collect(plan)` too, so that the `forall` at L2636 cannot pass on an empty list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4145449785
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -151,6 +151,63 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
}
}
+ /**
+ * The conjuncts of `afterScanFilters` the file format can evaluate at the storage layer for late
+ * materialization, to prune the IO of the columns the filter does not reference.
+ *
+ * They stay in the post-scan `Filter` as well, the way a pushed data filter does. The reader is
+ * offered them, not obliged to honor them, so the plan keeps the exact check. What it costs is
+ * evaluating the conjunct a second time for the rows the scan emits. What it buys is a reader
+ * free to give up on a file it cannot read in part, or on a stretch of rows whose survivors cost
+ * too much to keep track of, without the answer depending on it.
+ *
+ * Two of the conditions are per scan, and failing either offers nothing:
+ * - [[FileFormat.supportsStorageFilterPushdown]] holds. That is where a format reads the conf
+ * that enables this, so a format's own conf never decides for another format, and asking it
+ * first keeps everything below off the path of a scan that will not use it.
+ * - [[FileFormat.supportBatch]] holds for the schema the reader will see,
+ * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives
+ * its own vectorized-read decision from. Late materialization needs a batch read, so this asks
+ * about batch support rather than naming a format.
+ *
+ * The rest are per conjunct:
+ * - It is deterministic. The reader drops the rows the conjunct rejects and the post-scan
+ * `Filter` evaluates it again on the rows it keeps, so the two evaluations have to agree.
+ * - It references at least one column, and every column it references is a projected data
+ * column. A reference to something the scan does not read cannot be evaluated by the reader.
+ * - [[FileFormat.supportsStorageFilter]] accepts it. That is where the expression shapes and
+ * column types a reader can evaluate live, so this method names neither a format nor a type.
+ *
+ * One last condition is on the set that survives, that at least one projected column is left for
+ * the reader to prune. A scan that projects nothing but the filter's own storage-key columns
+ * reads the same columns for the same rows either way, since the reader has to read a key column
+ * to evaluate the filter on it, so offering it could only add the cost of evaluating the
+ * predicate outside the generated code. Nothing is offered in that case.
+ */
+ private def storageFiltersFor(
+ afterScanFilters: ExpressionSet,
+ fsRelation: HadoopFsRelation,
+ readDataColumns: Seq[Attribute],
+ outputDataSchema: StructType): Seq[Expression] = {
+ val sparkSession = fsRelation.sparkSession
+ if (!fsRelation.fileFormat.supportsStorageFilterPushdown(sparkSession)) return Nil
+ val resultSchema = StructType(fsRelation.partitionSchema.fields ++ outputDataSchema.fields)
+ if (!fsRelation.fileFormat.supportBatch(sparkSession, resultSchema)) return Nil
+
+ val dataAttrs = AttributeSet(readDataColumns)
+ // Over `toSeq` rather than the set, because `ExpressionSet.filter` hands the predicate
+ // `e.canonicalized`, which drops attribute names and metadata, and a format deciding by either
+ // would answer about an expression it will never be given.
+ val offered = afterScanFilters.toSeq.filter { expr =>
Review Comment:
Fixed with `normalizeExprs`. The offered conjuncts go through `DataSourceStrategy.normalizeExprs(_, readDataColumns)` before anything asks about them, so the format's gate and `create` see the same name, the relation's. It also means the `StorageFilters` entry in the scan description shows the relation's names, as `DataFilters` already does.
A new test plans a bloom over `_TMP_METADATA_ROW_INDEX` against a file whose column is `_tmp_metadata_row_index`, and asserts nothing is offered. Its control arm plans one over `K` and asserts it is offered under `k`. Without the normalization it fails.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled when the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ *
+ * <p>That happens only before the first row group is loaded. The plain path reads through
+ * parquet's own row-group cursor, and the phases below read row groups by index, which never
+ * advances it. So nulling this later would restart the read at the first row group.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch's key slots hold the head of each survivor queue, not the persistent key vectors, so
+ // closing that batch would leave those unreached. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, and the plain read path skips them too. Past here one
+ // would reach `pushedFilterRangesFor`, whose `RowRanges.createSingle(0)` under
+ // `pageIndexDisabled` builds Range(0, -1) and trips its `from <= to` assertion.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Neither depends on the columns
+ // requested, so giving the filter up would only meet the same null in phase 2.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ long keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // The same ranges as the baseline when nothing was pruned, and then the walk would arrive
+ // at the same number a second time.
+ long phase2Bytes = finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += blockAccounting.compressedBytesForRowRanges(blockIdx,
+ keyOnlyPaths, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ long filteredRows = baselineRows - keptRows;
+ if (filteredRows > 0) m.rowsExcludedWithinRowGroup().add(filteredRows);
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ RowRanges readerRanges = readerRangesFor(finalRanges, finalRowCount, blockRowCount);
+ installRowGroup(dataPages, readerRanges, keptRows);
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queues at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ return spliceCurrentRowGroup && isKeyTopLevel[slot];
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses, since the rows its column index already excluded were never this filter's
+ // to save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ *
+ * <p>A file this reader can no longer read in part still comes through here, since these are the
+ * rows a plain read of the same query would read, and leaving them to the post-scan Filter
+ * instead would make the scan slower than having no storage filter at all.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * What the surviving rows of the current row group cost to hold as row ranges, which is the half
+ * of the budget that a row group buffering nothing still pays. A filter whose survivors are
+ * scattered makes one range per surviving row, and phase 2 needs the whole set to select its
+ * pages. One set is held, not one per column reader, and the readers of a row group walk that
+ * same list, each with a cursor of its own.
+ */
+ private static long rowRangeStateBytes(long rangeCount) {
+ // Parquet's own `RowRanges.Range` holds two longs, plus its object header and a list slot.
+ return rangeCount * 40L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group. The
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ ensureKeyScratchAllocated();
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below, which is the bytes buffered
+ // for splicing and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so whether anything survives
+ // at all was the only thing left to learn, and it does. Stop here, since the ranges are
+ // not needed and the rest of the key column need not be decoded or evaluated. The
+ // row groups the filter empties are still skipped, which is the case that returns a
+ // non-null empty range set below.
+ abandonSplicing();
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) splicedBytes += appendSurvivorRowToAccumulators(r);
+ // Both halves of what this row group retains grow per survivor, and either can cross
+ // the budget on its own, so they are weighed together here and nowhere else. The cheaper
+ // concession comes first, which is to release the buffer, and the filter goes too if the
+ // ranges alone still do not fit.
+ long rangeBytes = rowRangeStateBytes(survivorRangeCount);
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper concession first, releasing the buffer, which is a no-op for a row group
+ // that has already given splicing up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ if (rangeBytes > cap) {
+ // The ranges being built are about to be thrown away, so stop evaluating the rest.
+ return null;
+ }
+ }
+ } catch (Exception e) {
Review Comment:
Fixed in all three catches, the two you name and the new decode one below, with the checks Spark already makes rather than new ones. A pending kill is read off `TaskContext.isInterrupted()`, the way `PythonRunner` reads one. A fatal error is found by `Executor.isFatalError`, as deep as `spark.executor.killOnFatalError.depth` says, read the way `SparkUncaughtExceptionHandler` reads it. Either is rethrown as it is, not as a `TaskKilledException`, for the `ignoreCorruptFiles` reason you give. The check lives in `ParquetStorageFilter.rethrowIfMustPropagate`, and all three catches call it first.
A new test covers both. One arm marks a task context interrupted and asserts the key expression's checked `SparkException` reaches the caller. The other has the expression wrap an `OutOfMemoryError` and asserts the same. Making the check always answer no fails it.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled when the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ *
+ * <p>That happens only before the first row group is loaded. The plain path reads through
+ * parquet's own row-group cursor, and the phases below read row groups by index, which never
+ * advances it. So nulling this later would restart the read at the first row group.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch's key slots hold the head of each survivor queue, not the persistent key vectors, so
+ // closing that batch would leave those unreached. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, and the plain read path skips them too. Past here one
+ // would reach `pushedFilterRangesFor`, whose `RowRanges.createSingle(0)` under
+ // `pageIndexDisabled` builds Range(0, -1) and trips its `from <= to` assertion.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Neither depends on the columns
+ // requested, so giving the filter up would only meet the same null in phase 2.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ long keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // The same ranges as the baseline when nothing was pruned, and then the walk would arrive
+ // at the same number a second time.
+ long phase2Bytes = finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += blockAccounting.compressedBytesForRowRanges(blockIdx,
+ keyOnlyPaths, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ long filteredRows = baselineRows - keptRows;
+ if (filteredRows > 0) m.rowsExcludedWithinRowGroup().add(filteredRows);
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ RowRanges readerRanges = readerRangesFor(finalRanges, finalRowCount, blockRowCount);
+ installRowGroup(dataPages, readerRanges, keptRows);
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queues at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ return spliceCurrentRowGroup && isKeyTopLevel[slot];
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses, since the rows its column index already excluded were never this filter's
+ // to save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ *
+ * <p>A file this reader can no longer read in part still comes through here, since these are the
+ * rows a plain read of the same query would read, and leaving them to the post-scan Filter
+ * instead would make the scan slower than having no storage filter at all.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * What the surviving rows of the current row group cost to hold as row ranges, which is the half
+ * of the budget that a row group buffering nothing still pays. A filter whose survivors are
+ * scattered makes one range per surviving row, and phase 2 needs the whole set to select its
+ * pages. One set is held, not one per column reader, and the readers of a row group walk that
+ * same list, each with a cursor of its own.
+ */
+ private static long rowRangeStateBytes(long rangeCount) {
+ // Parquet's own `RowRanges.Range` holds two longs, plus its object header and a list slot.
+ return rangeCount * 40L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group. The
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ ensureKeyScratchAllocated();
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below, which is the bytes buffered
+ // for splicing and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
Review Comment:
Done, and your reasoning holds. Phase 1's `readBatch` is inside the fail-open now, with its own WARN. Giving the row group up hands it to phase 2 whole, over `pushedFilterRanges`, so the corrupt page fails at the same batch a plain read fails at, after the same rows. That was also where my round-4 answer went wrong. The difference came from where the catch boundary was drawn, not from evaluating before the first batch.
A new test writes one row group of 100-row pages, damages the body of the fifth key page, and reads it with both readers. Both must fail the same way after the same 400 rows. Rethrowing from the new catch instead leaves the filtering read with none.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1947,6 +1947,52 @@ object SQLConf {
.checkValue(threshold => threshold >= 0, "The threshold must not be negative.")
.createWithDefault(10)
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED =
+ buildConf("spark.sql.parquet.storageFilterPushdown.enabled")
+ .doc("If true, the vectorized Parquet reader may apply a runtime storage filter, such as a " +
+ "bloom filter from join runtime filtering, while it reads. It reads the columns the " +
+ "filter needs first, evaluates the filter per row, and then reads the remaining columns " +
+ "only for the rows that survived. This is a planning-time decision. With it false, no " +
+ "storage filter is attached to a scan at all. A filter that is attached also stays in " +
+ "the post-scan filter, the way a pushed data filter does, so the reader is free to stop " +
+ "applying it wherever doing so would cost more than it saves, and the answer does not " +
+ "change. A row group where it stops reads its key columns twice, which is slower than " +
+ "not pushing the filter. Under spark.sql.files.ignoreCorruptFiles the answer can change. " +
+ "A corrupt page in a key column then also drops the rows of its row group that a plain " +
Review Comment:
Rewritten in general terms. The conf doc says that under `ignoreCorruptFiles` this reader reads different pages in a different order than a plain read, so which rows survive a corrupt page can differ from a plain read in either direction. The description says the same, with your two cases, and that a corrupt key page is no longer one of them after the change above.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled when the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ *
+ * <p>That happens only before the first row group is loaded. The plain path reads through
+ * parquet's own row-group cursor, and the phases below read row groups by index, which never
+ * advances it. So nulling this later would restart the read at the first row group.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch's key slots hold the head of each survivor queue, not the persistent key vectors, so
+ // closing that batch would leave those unreached. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
Review Comment:
Fixed as you suggest. The decline counts only the non-key fields whose `ParquetColumn` is not in `missingColumns`, so the row-index column drops out too. A new test evolves a table by a non-key column and reads `k, w` with a storage filter on `k`. The older files come back unfiltered and the newer ones filtered, and counting every field again fails it.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled when the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ *
+ * <p>That happens only before the first row group is loaded. The plain path reads through
+ * parquet's own row-group cursor, and the phases below read row groups by index, which never
+ * advances it. So nulling this later would restart the read at the first row group.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch's key slots hold the head of each survivor queue, not the persistent key vectors, so
+ // closing that batch would leave those unreached. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file, which then yields no rows. True, or an error, reads it the way a plain
+ * scan would, under the same fail-open rule as a row's value, since the constant can be one the
+ * predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (Exception e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ totalRowCount = 0;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, and the plain read path skips them too. Past here one
+ // would reach `pushedFilterRangesFor`, whose `RowRanges.createSingle(0)` under
+ // `pageIndexDisabled` builds Range(0, -1) and trips its `from <= to` assertion.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Neither depends on the columns
+ // requested, so giving the filter up would only meet the same null in phase 2.
+ throw new IllegalStateException(
+ "No key pages for row group " + blockIdx + " despite " + baselineRows
+ + " rows selected by the pushed filter");
+ }
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ long keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up. It read what a plain scan
+ // reads plus the key columns phase 1 had already read, so there is no saving to report.
+ if (!filterGivenUp) {
+ // The same ranges as the baseline when nothing was pruned, and then the walk would arrive
+ // at the same number a second time.
+ long phase2Bytes = finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += blockAccounting.compressedBytesForRowRanges(blockIdx,
+ keyOnlyPaths, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ long filteredRows = baselineRows - keptRows;
+ if (filteredRows > 0) m.rowsExcludedWithinRowGroup().add(filteredRows);
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ RowRanges readerRanges = readerRangesFor(finalRanges, finalRowCount, blockRowCount);
+ installRowGroup(dataPages, readerRanges, keptRows);
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queues at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ return spliceCurrentRowGroup && isKeyTopLevel[slot];
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses, since the rows its column index already excluded were never this filter's
+ // to save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ *
+ * <p>A file this reader can no longer read in part still comes through here, since these are the
+ * rows a plain read of the same query would read, and leaving them to the post-scan Filter
+ * instead would make the scan slower than having no storage filter at all.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * What the surviving rows of the current row group cost to hold as row ranges, which is the half
+ * of the budget that a row group buffering nothing still pays. A filter whose survivors are
+ * scattered makes one range per surviving row, and phase 2 needs the whole set to select its
+ * pages. One set is held, not one per column reader, and the readers of a row group walk that
+ * same list, each with a cursor of its own.
+ */
+ private static long rowRangeStateBytes(long rangeCount) {
+ // Parquet's own `RowRanges.Range` holds two longs, plus its object header and a list slot.
+ return rangeCount * 40L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group. The
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ ensureKeyScratchAllocated();
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below, which is the bytes buffered
+ // for splicing and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so whether anything survives
+ // at all was the only thing left to learn, and it does. Stop here, since the ranges are
+ // not needed and the rest of the key column need not be decoded or evaluated. The
+ // row groups the filter empties are still skipped, which is the case that returns a
+ // non-null empty range set below.
+ abandonSplicing();
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) splicedBytes += appendSurvivorRowToAccumulators(r);
+ // Both halves of what this row group retains grow per survivor, and either can cross
+ // the budget on its own, so they are weighed together here and nowhere else. The cheaper
+ // concession comes first, which is to release the buffer, and the filter goes too if the
+ // ranges alone still do not fit.
+ long rangeBytes = rowRangeStateBytes(survivorRangeCount);
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper concession first, releasing the buffer, which is a no-op for a row group
+ // that has already given splicing up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ if (rangeBytes > cap) {
+ // The ranges being built are about to be thrown away, so stop evaluating the rest.
+ return null;
+ }
+ }
+ } catch (Exception e) {
+ // Fail open, whatever the exception is and wherever in the row it arose. The predicate
+ // ran on a row that, in the plan, an earlier conjunct would have rejected before it, so a
+ // plain scan never evaluates it there. Giving the filter up for this row group puts every
+ // row the pushed filter kept back in the output, and the post-scan Filter then evaluates
+ // the conjuncts in their own order. Either an earlier one drops the row before this
+ // expression runs, or it does not and the query fails the way it would have without this
+ // feature.
+ //
+ // Not narrowed to Spark's own errors, because a built-in expression can raise a plain JDK
+ // one, as `timestamp_seconds` of a decimal throws ArithmeticException from
+ // longValueExact. Nor to unchecked ones, because a Hive UDF in the key throws a checked
+ // SparkException (FAILED_EXECUTE_UDF) that nothing declares. Rethrowing has two bad ends,
+ // and neither is the error the plan would have raised. The query fails where a plain read
+ // succeeds, and under `ignoreCorruptFiles` this reader's exception is read as a corrupt
+ // file, which drops the rest of it silently.
+ //
+ // Copying a surviving key lands here too, which a survivor vector that cannot grow does,
+ // and reading the row group with nothing buffered is the answer to that as well.
+ logFilterGivenUpOnError(e);
+ abandonSplicing();
+ return null;
+ }
+ }
+ remaining -= num;
+ }
+
+ // Any partially-filled accumulator goes to its queue, so the emit path can dequeue it as this
+ // row group's final batch.
+ if (spliceCurrentRowGroup && currentKeyAccumulatorRowCount > 0) {
+ pushAccumulatorsToQueues();
+ }
+
+ return finalRangesBuilder.build();
+ }
+
+ private void ensureKeyScratchAllocated() {
+ if (keyScratchVectors != null) return;
+ // Assigned before the loop on purpose, so that an allocation failure part way through leaves
+ // the vectors allocated so far reachable for `close()`, which walks this array element-wise.
+ keyScratchVectors = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyScratchVectors[i] = newKeyVector(i);
+ }
+ // The row the predicate evaluates holds the key columns phase 1 reads, and for a key column
+ // this file does not have, the constant vector `initBatch` built for it. `reset()` is a no-op
+ // on those, and this reader never closes them, since the emitted batch owns them.
+ ColumnVector[] rowVectors = keyVectorsFromBatch();
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ rowVectors[keyColumns[i].rowPosition()] = keyScratchVectors[i];
+ }
+ keyScratchBatch = new ColumnarBatch(rowVectors);
+ }
+
+ /**
+ * The key columns' vectors as the emitted batch holds them, in the order the predicate's row
+ * reads them. Which is what a missing key column reads as, and what phase 1 overwrites for a key
+ * column this file has. Taking a present key's batch vector and discarding it is a getter over a
+ * final field, and it saves holding the missing positions as state.
+ */
+ private ColumnVector[] keyVectorsFromBatch() {
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ ColumnVector[] vectors = new ColumnVector[keyIndices.length];
+ for (int p = 0; p < keyIndices.length; p++) {
+ vectors[p] = columnVectors[keyIndices[p]].getValueVector();
+ }
+ return vectors;
+ }
+
+ /** A capacity-sized vector for the i-th key column this file has, in the reader's memory mode. */
+ private WritableColumnVector newKeyVector(int keyIdx) {
+ DataType dt = keyColumns[keyIdx].type();
+ return MEMORY_MODE == MemoryMode.OFF_HEAP
+ ? new OffHeapColumnVector(capacity, dt)
+ : new OnHeapColumnVector(capacity, dt);
+ }
+
+ /**
+ * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows
+ * each.
+ */
+ private void ensureCurrentKeyAccumulatorsAllocated() {
+ for (int i = 0; i < currentKeyAccumulators.length; i++) {
+ if (currentKeyAccumulators[i] == null) {
+ currentKeyAccumulators[i] = newKeyVector(i);
Review Comment:
Fixed the cheap way. When the next accumulator of a variable-length key is allocated, its byte child is `reserve`d to what the set just queued held for that key, `previous[i].arrayData().getElementsAppended()`. The first accumulator of a row group still starts from the default, since its queue is empty then. That memory is taken before any value arrives, so the budget now charges it when the accumulator is sized, by the same factor of four, and the values written into it use that charge up before they add to it. Charged only as values arrived, the last accumulator of a row group would have held a whole accumulator's worth for a few survivors' charge. A new test has six 1000-byte keys over a capacity of 4 under a cap only the up-front charge crosses, and without that charge it keeps splicing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4145455892
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -476,34 +506,80 @@ private boolean containsPath(Type parquetType, String[] path, int depth) {
return false;
}
- private void checkEndOfRowGroup() throws IOException {
- if (rowsReturned != totalCountLoadedSoFar) return;
+ /**
+ * Loads the row group the read continues with, and returns false if there is none left. Only a
+ * subclass that picks its own rows can answer false, because this class reads every row of every
+ * row group, and then {@code totalRowCount} already says when they run out.
+ */
+ protected boolean loadNextRowGroup() throws IOException {
PageReadStore pages = reader.readNextRowGroup();
if (pages == null) {
throw new IOException("expecting more rows but reached last block. Read "
+ rowsReturned + " out of " + totalRowCount);
}
+ // Null ranges, because this path never picks the rows itself, and whether parquet narrowed the
+ // row group by a pushed filter's column index is known only to the store it handed back.
+ installRowGroup(pages, null, pages.getRowCount());
+ return true;
+ }
+
+ /**
+ * Points the column readers at one row group's {@code pages}, from which they are to read the
+ * rows {@code rowRanges} names, or every row of the row group when it is null. {@code rowCount}
+ * is how many rows that is, which is what the read has left before it needs another row group.
+ */
+ protected final void installRowGroup(PageReadStore pages, RowRanges rowRanges, long rowCount)
Review Comment:
Removed. `installRowGroup` reads `pages.getRowCount()` itself, as master did, and the assertion from your store-count item is what keeps the subclass's `finalRowCount` equal to it.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -80,7 +85,7 @@ public class VectorizedParquetRecordReader extends SpecificParquetRecordReaderBa
* Encapsulate writable column vectors with other Parquet related info such as
* repetition / definition levels.
*/
- private ParquetColumnVector[] columnVectors;
+ protected ParquetColumnVector[] columnVectors;
Review Comment:
Both done, and taken further. `keyVectorsFromBatch` reads `persistentBatchColumns`, and `columnVectors` is private again. The `resetBatch` and `readBatchColumns` hooks are gone as well as `finishBatch`, so the base's `nextBatch` is master's bar the line that calls the load. The subclass overrides `nextBatch` whole and points the key slots after the base has read. `closeAllocated` went the same way, so the base's `close` is master's too. The base class has six protected methods and four protected fields now.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -0,0 +1,2755 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.{File, RandomAccessFile}
+import java.net.URI
+import java.time.LocalTime
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem}
+import org.apache.hadoop.mapred.FileSplit
+import org.apache.hadoop.mapreduce.{Job, TaskAttemptID}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.parquet.column.{Encoding, ParquetProperties}
+import org.apache.parquet.column.impl.ColumnWriteStoreV1
+import org.apache.parquet.column.page.DataPageV1
+import org.apache.parquet.column.page.mem.MemPageStore
+import org.apache.parquet.format.converter.ParquetMetadataConverter
+import org.apache.parquet.hadoop.{ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat}
+import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName, ParquetMetadata}
+import org.apache.parquet.hadoop.util.{HadoopInputFile, HadoopOutputFile}
+import org.apache.parquet.schema.MessageTypeParser
+
+import org.apache.spark.SparkException
+import org.apache.spark.paths.SparkPath
+import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, SecondsToTimestamp, UnaryExpression, XxHash64}
+import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter}
+import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.Utils
+import org.apache.spark.util.sketch.BloomFilter
+
+/**
+ * Tests [[LateMaterializationParquetRecordReader]] driven by a [[ParquetStorageFilter]]. Writes
+ * small multi-row-group parquet files, wires a hand-built filter into the reader, and asserts
+ * correctness and the five storage-filter metrics.
+ */
+class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
+ with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write
+ // helper in this suite goes through here.
+ private def writeSingleParquetFile(
+ dir: File,
+ df: DataFrame,
+ rowGroupSize: Long,
+ pageSize: Option[Long] = None,
+ dictionary: Boolean = false): String = {
+ val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath
+ val writer = df
+ .repartition(1)
+ .write
+ .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize)
+ // Dictionary encoding off keeps row-group sizing predictable. The column index is still
+ // written either way.
+ .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString)
+ // A small page size gives each row group several pages per column, which is what lets
+ // column-index filtering produce a row range narrower than the whole row group.
+ pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size))
+ writer.parquet(outDir)
+ val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet"))
+ assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir")
+ files(0).getAbsolutePath
+ }
+
+ // Writes a parquet file with the given rows and row-group size; returns the path.
+ private def writeParquetFile(
+ dir: File,
+ rows: Seq[(Long, String)],
+ rowGroupSize: Long = 1024L,
+ pageSize: Option[Long] = None): String =
+ writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize)
+
+ // The scan node of a plan, which every test that inspects or rewrites `storageFilters` needs.
+ private def scanOf(plan: SparkPlan): FileSourceScanExec =
+ plan.collect { case s: FileSourceScanExec => s }.headOption
+ .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan"))
+
+ // A hadoop conf whose reads go through `CountingLocalFileSystem`, so a test can measure them.
+ private def countingHadoopConf(): Configuration = {
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl",
+ classOf[CountingLocalFileSystem].getName)
+ hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true)
+ hadoopConf
+ }
+
+ // Reads `path` whole through `readerFn`, and returns the rows it emitted with the bytes the read
+ // transferred. The reader must have been built with `countingHadoopConf`.
+ private def readCounting(
+ path: String, readerFn: PartitionedFile => Iterator[_]): (Int, Long) = {
+ val file = PartitionedFile(
+ InternalRow.empty,
+ SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"),
+ 0,
+ new File(path).length())
+ CountingLocalFileSystem.reset()
+ val emitted = readerFn(file).asInstanceOf[Iterator[Object]].map {
+ case batch: ColumnarBatch => batch.numRows()
+ case _ => 1
+ }.sum
+ (emitted, CountingLocalFileSystem.bytesRead())
+ }
+
+ // The metric map a scan hands the format, for a test that builds a reader by hand.
+ private def metricMap(): Map[String, SQLMetric] =
+ new ParquetFileFormat().storageFilterMetrics(spark.sparkContext)
+
+ // Every metric the reader updates, created the way the production path creates them. A test that
+ // asserts one holds the whole set and reads the counter off it.
+ private def allMetrics(): StorageFilterMetrics =
+ StorageFilterMetrics.create(spark.sparkContext)
+
+ // `ParquetStorageFilter.create` takes what the production path always supplies; most tests care
+ // about neither the metrics nor the cap.
+ private def createFilter(
+ boundExpressions: Seq[Expression],
+ requestedSchema: StructType,
+ metrics: StorageFilterMetrics = allMetrics(),
+ maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter =
+ ParquetStorageFilter.create(
+ boundExpressions, requestedSchema, metrics, maxSplicedRowGroupBytes)
+
+ // Collects all `(k, v)` rows from a reader initialized with the given storage filter.
+ private def readAll(
+ filePath: String,
+ storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString))
+
+ // Builds a `k >= threshold` storage filter bound to position 0, over a long key.
+ private def keyAtLeastFilter(
+ threshold: Long,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter =
+ keyTypeAtLeastFilter(Literal(threshold), LongType, metrics)
+
+ // A key column of the caller's type plus one non-key column, written via Spark's `Encoder`. Every
+ // key-type fixture carries that second column, because a projection of key columns alone is one
+ // the reader declines. It would read what a plain scan reads, so there is nothing for phase 2 to
+ // prune.
+ private def writeKeyParquetFile[T : org.apache.spark.sql.Encoder](
+ dir: File,
+ keys: Seq[T],
+ rowGroupSize: Long = 1024L): String =
+ writeSingleParquetFile(
+ dir,
+ spark.createDataset(keys).toDF("k").selectExpr("k", "CAST(k AS STRING) AS v"),
+ rowGroupSize)
+
+ // Reads a key-type fixture, returning the survivor keys and the reader. The `extract` function
+ // pulls one value at a time from the batch's key column.
+ private def readKeysAll[T](
+ filePath: String,
+ storageFilter: ParquetStorageFilter,
+ extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T,
+ capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) =
+ readAllWith(filePath, Seq("k", "v"), storageFilter, (batch, i) => extract(batch.column(0), i),
+ capacity)
+
+ // Builds a `k >= threshold` storage filter bound to position 0 against a `(k, v)` schema whose
+ // key has the given type.
+ private def keyTypeAtLeastFilter(
+ threshold: Literal,
+ keyType: DataType,
+ metrics: StorageFilterMetrics = allMetrics()): ParquetStorageFilter = {
+ val expr = GreaterThanOrEqual(BoundReference(0, keyType, nullable = false), threshold)
+ val requested = StructType(Seq(
+ StructField("k", keyType, nullable = false), StructField("v", StringType, nullable = true)))
+ createFilter(Seq(expr), requested, metrics)
+ }
+
+ test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") {
+ withTempDir { dir =>
+ // 40 rows come out as one row group. Parquet's first row-group size check is at record 100,
+ // so `rowGroupSize` cannot split a file this small. The one row group is the one rejected.
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(1000L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.isEmpty, "filter rejects all rows; no rows should be emitted")
+ assert(m.rowGroupsSkipped.value > 0,
+ s"expected at least one row group skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedByRowGroup.value > 0,
+ s"expected rows excluded by whole-rowgroup skip; got ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"no partial-row-group filtering expected; got ${m.rowsExcludedWithinRowGroup.value}")
+ // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the
+ // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0.
+ assert(m.bytesAvoidedByRowGroup.value > 0,
+ s"expected non-key bytes avoided by whole row groups; got " +
+ s"${m.bytesAvoidedByRowGroup.value}")
+ assert(m.bytesAvoidedByPageFiltering.value == 0,
+ s"no page-filtering bytes expected when all groups skipped; got " +
+ s"${m.bytesAvoidedByPageFiltering.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("all rows survive: no skipping and no filtering") {
+ withTempDir { dir =>
+ val rows = (1L to 40L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ val filter = keyAtLeastFilter(0L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows")
+ assert(m.rowGroupsSkipped.value == 0,
+ s"nothing should be skipped; got ${m.rowGroupsSkipped.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"nothing should be filtered; got ${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("mixed: some row groups skipped, others partially kept") {
+ withTempDir { dir =>
+ // Many rows + small row groups => guaranteed multiple row groups.
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ val m = allMetrics()
+ // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped.
+ val filter = keyAtLeastFilter(195L, m)
+ val (result, reader) = readAll(path, filter)
+ try {
+ // Output is exact. VectorizedColumnReader uses PageReadStore.getRowIndexes (driven by
+ // our finalRanges) to skip rows within partial pages, so emitted rows == survivors.
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(result.toSet == expected,
+ s"expected exact filtering; got ${result.map(_._1).sorted}, " +
+ s"expected ${expected.map(_._1).toSeq.sorted}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected row groups skipped; got ${m.rowGroupsSkipped.value}")
+ // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only
+ // this identity establishes it. A row group that was neither skipped whole nor emitted has
+ // to have its rows counted there.
+ val excludedRg = m.rowsExcludedByRowGroup.value
+ val excludedPf = m.rowsExcludedWithinRowGroup.value
+ assert(excludedPf > 0, s"expected rows excluded inside a kept row group; got $excludedPf")
+ assert(result.size + excludedRg + excludedPf == rows.size,
+ s"${result.size} emitted plus $excludedRg plus $excludedPf should account for all " +
+ s"${rows.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("an all-keys projection: the reader declines the filter and reads plainly") {
+ // A scan whose projected columns are all key columns of the filter reads exactly what a plain
+ // scan reads, because phase 1 has to read a key column to evaluate the filter on it, and
+ // phase 2 is then left with nothing to prune. All the filter can add there is cost, so the
+ // reader declines it rather than carrying a second emit path. The planner declines that shape
+ // too, which is why this is asserted here, against the reader driven directly.
+ withTempDir { dir =>
+ val keys = (1L to 200L)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(195L))
+ val metrics = allMetrics()
+ val filter = createFilter(Seq(expr), requested, metrics)
+ val (result, reader) = readAllWith(
+ path, Seq("k"), filter, (batch, i) => batch.column(0).getLong(i))
+ try {
+ assert(result == keys.toSeq,
+ s"every row of the file must come back unfiltered; got ${result.size} of ${keys.size}")
+ val reported = Seq(metrics.rowGroupsSkipped, metrics.rowsExcludedByRowGroup,
+ metrics.rowsExcludedWithinRowGroup, metrics.bytesAvoidedByRowGroup,
+ metrics.bytesAvoidedByPageFiltering).map(_.value)
+ assert(reported.forall(_ == 0L), s"and the filter must report nothing: $reported")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("Hadoop's 2-arg initialize initializes the reader once") {
+ // That overload delegates to the 5-arg one, which runs the reader's own initialization, and it
+ // used to run that a second time on top. A reader that declined the filter on the first pass,
+ // here because every projected column is a key column, then reached the filter it had just
+ // released and failed with an NPE instead of reading plainly.
+ withTempDir { dir =>
+ val keys = (1L to 20L)
+ val path = writeKeyParquetFile(dir, keys)
+ val requested = StructType(Seq(StructField("k", LongType, nullable = false)))
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(15L))),
+ requested)
+ // What `ParquetFileFormat` sets up for the read, down to what the schema converter needs.
+ val conf = spark.sessionState.newHadoopConf()
+ conf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ conf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, requested.json)
+ Seq(SQLConf.CASE_SENSITIVE, SQLConf.PARQUET_BINARY_AS_STRING,
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP, SQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED,
+ SQLConf.LEGACY_PARQUET_NANOS_AS_LONG).foreach { entry =>
+ conf.set(entry.key, spark.sessionState.conf.getConfString(entry.key))
+ }
+ val split = new FileSplit(new Path(path), 0, new File(path).length(), Array.empty[String])
+ val context = new TaskAttemptContextImpl(conf, new TaskAttemptID())
+ Utils.tryWithResource(new LateMaterializationParquetRecordReader(false, 4096, filter)) {
+ reader =>
+ reader.initialize(split, context)
+ reader.initBatch(new StructType(), null)
+ val read = mutable.ArrayBuffer[Long]()
+ while (reader.nextBatch()) {
+ val batch = reader.resultBatch()
+ (0 until batch.numRows()).foreach(i => read += batch.column(0).getLong(i))
+ }
+ assert(read == keys, s"every row must come back unfiltered; got ${read.size}")
+ }
+ }
+ }
+
+ test("multi-batch emit: survivor count exceeds capacity") {
+ // Drive the reader at capacity = 16 with a row group of 71 survivors out of 100. Exercises:
+ // - The per-key-column queue holding multiple full-capacity vectors plus a partial tail.
+ // - The published queue head getting closed at the start of every subsequent emit.
+ // - The batch's key slots being rewritten ceil(71/16) = 5 times.
+ // The filter keeps most of the row group rather than all of it, so a reader that declined the
+ // filter altogether would return more rows than this asserts rather than passing.
+ withTempDir { dir =>
+ val keys = (1L to 100L)
+ // Big rowGroupSize so all 100 rows fit in one row group.
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 64 * 1024L)
+ val filter = keyTypeAtLeastFilter(Literal(30L), LongType)
+ val (result, reader) =
+ readKeysAll(path, filter, (vec, i) => vec.getLong(i), capacity = 16)
+ try {
+ assert(result == keys.filter(_ >= 30L).toSeq,
+ s"expected the surviving keys in order across multiple batches; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("int key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the IntegerType branch of ValueCopier.
+ withTempDir { dir =>
+ val keys = (1 to 100)
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyTypeAtLeastFilter(Literal(90), IntegerType)
+ val (result, reader) = readKeysAll(path, filter, (vec, i) => vec.getInt(i))
+ try {
+ assert(result.toSet == keys.filter(_ >= 90).toSet,
+ s"expected int keys >= 90; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("string key column: filter survivors round-trip through phase 1 accumulators") {
+ // Covers the StringType branch of ValueCopier (variable-length byte copy via putByteArray).
+ withTempDir { dir =>
+ val keys = (1 to 20).map(i => f"k$i%03d")
+ val path = writeKeyParquetFile(dir, keys, rowGroupSize = 256L)
+ val filter = keyTypeAtLeastFilter(
+ Literal.create("k015", StringType), StringType)
+ val (result, reader) =
+ readKeysAll(path, filter, (vec, i) => vec.getUTF8String(i).toString)
+ try {
+ assert(result.toSet == keys.filter(_ >= "k015").toSet,
+ s"expected string keys >= 'k015'; got ${result.sorted}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") {
+ // These are all planner bugs by construction, because storageFiltersFor pre-checks each one.
+ // The rows are safe either way, since the conjunct stays in the post-scan Filter, but a soft
+ // rejection would hide the bug behind a silently slower read. create fails instead.
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+
+ // Nothing to push. The caller is supposed to check this before calling.
+ val empty = intercept[IllegalArgumentException] {
+ createFilter(Seq.empty, requested)
+ }
+ assert(empty.getMessage.contains("must be non-empty"), empty.getMessage)
+
+ // Ordinal 5 is out of range for a two-field requested schema.
+ val outOfRange = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))),
+ requested)
+ }
+ assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage)
+
+ // No bound reference at all, so there is no key column to read in phase 1.
+ val noRefs = intercept[IllegalArgumentException] {
+ createFilter(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested)
+ }
+ assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage)
+
+ // A key type the reader has no value copier for.
+ val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true)))
+ val badType = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema)
+ }
+ assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage)
+
+ // A key column named like the synthetic row-index metadata column. This reader finds that
+ // column by name and writes row indexes over whatever the file holds, so what a key column of
+ // that name reads back would depend on how its row group was read. Both gates reject it. The
+ // planner's means the filter is never offered, and this one means a caller past the planner
+ // fails rather than reading a key the reader itself overwrote.
+ val rowIndexName = ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+ val rowIndexSchema = StructType(Seq(
+ StructField(rowIndexName, LongType, nullable = false),
+ StructField("v", StringType, nullable = false)))
+ val rowIndexKey = intercept[IllegalArgumentException] {
+ createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(0L))),
+ rowIndexSchema)
+ }
+ assert(rowIndexKey.getMessage.contains(rowIndexName), rowIndexKey.getMessage)
+ val rowIndexAttr = AttributeReference(rowIndexName, LongType, nullable = false)()
+ assert(!ParquetStorageFilter.isSupportedStorageFilter(
+ BloomFilterMightContain(
+ Literal(bloomBytes(BloomFilter.create(10))),
+ new XxHash64(Seq(rowIndexAttr)))),
+ "a bloom over a column named like the row-index column must not be offered")
+ }
+
+ // Serializes a [[BloomFilter]] to bytes suitable for a [[Literal]].
+ // The same serialization a runtime bloom's Literal carries, so these tests build theirs the way
+ // the planner does.
+ private def bloomBytes(bf: BloomFilter): Array[Byte] = BloomFilterAggregate.serialize(bf)
+
+ // Computes `XxHash64(v)` using the same seed Spark uses for runtime bloom filters.
+ private def xxHash64(v: Any, dt: DataType): Long = {
+ new XxHash64(Seq(Literal(v, dt))).eval(InternalRow.empty).asInstanceOf[Long]
+ }
+
+ test("a plain JDK exception from the key expression gives the row group up") {
+ // The reader's fail-open is not narrowed to Spark's own errors, and this is why. A built-in
+ // expression that `InjectRuntimeFilter` is happy to build a bloom over can raise a plain JDK
+ // one. Rethrowing it would fail a query a plain read answers, and under `ignoreCorruptFiles`
+ // an exception from a reader is read as a corrupt file, which drops the rest of it silently.
+ //
+ // The filter is selective, so returning every row is what proves the give-up. A filter that
+ // ran to the end would have dropped the rows below 10 seconds.
+ withTempDir { dir =>
+ val df = spark.range(0, 20).selectExpr(
+ "CAST(CASE WHEN id = 7 THEN 1.0000001 ELSE id END AS DECIMAL(20,7)) AS k",
+ "CAST(id AS STRING) AS v")
+ val path = writeSingleParquetFile(dir, df, rowGroupSize = 64 * 1024L)
+ val decimalType = DecimalType(20, 7)
+ val requested = StructType(Seq(
+ StructField("k", decimalType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ // Row 7 throws, since 1.0000001 seconds is not a whole number of microseconds.
+ val secondsKey = GreaterThanOrEqual(
+ SecondsToTimestamp(BoundReference(0, decimalType, nullable = true)),
+ Literal.create(10L * 1000000L, TimestampType))
+ val filter = createFilter(Seq(secondsKey), requested)
+ val (result, reader) = readAllWith(
+ path, Seq("k", "v"), filter, (b, i) => b.column(1).getUTF8String(i).toString)
+ try {
+ assert(result == (0 until 20).map(_.toString),
+ s"the row group must come back whole; got ${result.size} rows: $result")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a checked exception from the key expression gives the row group up") {
+ // Nor is the fail-open narrowed to unchecked exceptions. A Hive UDF in the key wraps what it
+ // throws in a checked SparkException (FAILED_EXECUTE_UDF), which no signature declares, so a
+ // catch of RuntimeException alone would fail a query a plain read answers. The filter is
+ // selective, so returning every row is what proves the give-up.
+ withTempDir { dir =>
+ val rows = (0L until 20L).map(i => (i, i.toString))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L)
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", StringType, nullable = true)))
+ val key = ThrowsCheckedOn(BoundReference(0, LongType, nullable = false), bad = 7L)
+ val filter = createFilter(Seq(GreaterThanOrEqual(key, Literal(10L))), requested)
+ val (result, reader) = readAll(path, filter)
+ try {
+ assert(result == rows, s"the row group must come back whole; got ${result.size} rows")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ // Returns the FileSourceScanExec for a parquet read, with `storageFilters` attached. Goes through
+ // Spark's normal planning/execution machinery (not the test-only reader init), so it exercises
+ // preparedStorageFilters' subquery materialization + bind, the SQL conf check,
+ // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation.
+ private def scanWithStorageFilter(
+ path: String,
+ keyName: String,
+ threshold: Long): FileSourceScanExec = {
+ val df = spark.read.parquet(path).select("k", "v")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ val keyAttr = scan.output.find(_.name == keyName).getOrElse(fail(s"No $keyName in scan output"))
+ val expr = GreaterThanOrEqual(keyAttr, Literal(threshold))
+ scan.copy(storageFilters = Seq(expr))
+ }
+
+ // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output
+ // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a
+ // ColumnarToRowExec; since these tests bypass the planner, we wrap manually.
+ private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] = {
+ val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan
+ rowPlan.executeCollect().map(r => (r.getLong(0), r.getString(1)))
+ }
+
+ test("end-to-end via FileSourceScanExec: the filter is applied and the metrics populated") {
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, s"v_$i"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val scan = scanWithStorageFilter(path, "k", threshold = 195L)
+ val collected = executePlanCollect(scan).toSet
+ val expected = rows.filter(_._1 >= 195L).toSet
+ assert(collected == expected, s"got ${collected.toSeq.sortBy(_._1)}; expected $expected")
+
+ val rgSkipped = scan.metrics(StorageFilterMetrics.ROW_GROUPS_SKIPPED)
+ assert(rgSkipped.value >= 1,
+ s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}")
+ }
+ }
+ }
+
+ test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") {
+ // A pushed data filter on a non-key column and a storage filter on the key column have to
+ // compose. Phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the
+ // storage filter, and phase 2 reads the non-key columns under the intersection.
+ //
+ // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes
+ // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is
+ // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be
+ // satisfied by a phase 0 that saw the wrong schema, and would prove nothing.
+ withTempDir { dir =>
+ val rows = (1L to 200L).map(i => (i, f"v_$i%03d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+
+ withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'")
+ val plan = df.queryExecution.executedPlan
+ val scan = scanOf(plan)
+ assert(scan.simpleString(200).contains("GreaterThan(v,"),
+ s"the data filter must actually be pushed for this test to mean anything: " +
+ scan.simpleString(200))
+ // Storage filter on `k` (key column).
+ val keyAttr = scan.output.find(_.name == "k").get
+ val withSF = scan.copy(
+ storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(100L))))
+
+ val collected = executePlanCollect(withSF).toSet
+ // Every row satisfies v > 'v_000', so the storage filter alone decides the result.
+ val expected = rows.filter(_._1 >= 100L).toSet
+ assert(collected == expected,
+ s"got ${collected.toSeq.sortBy(_._1)}; expected ${expected.toSeq.sortBy(_._1)}")
+ }
+ }
+ }
+
+ // ----- FileSourceStrategy bloom-filter extraction -----
+
+ // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan.
+ // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an
+ // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped.
+ private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case f: FilterExec =>
+ f.condition.collect { case _: BloomFilterMightContain => 1 }.sum
+ }.sum
+ }
+
+ // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters.
+ private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = {
+ collect(plan) {
+ case s: FileSourceScanExec =>
+ s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum
+ }.sum
+ }
+
+ // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the
+ // application-side scan. Returns the executed plan and the query result for inspection. Tables
+ // are cleaned up automatically by the calling test (table names are passed through).
+ private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = {
+ val query =
+ """SELECT bf1.k, bf1.v
+ |FROM bf1 JOIN bf2 ON bf1.k = bf2.k
+ |WHERE bf2.v = 5
+ |""".stripMargin
+ val df = spark.sql(query)
+ (df.queryExecution.executedPlan, df.collect())
+ }
+
+ // Creates two parquet tables for join tests, a large `bf1` and a small `bf2` with a selective
+ // filter, and sets what it takes for the optimizer to build a bloom over them at all. The
+ // application side must look big enough to be worth one, and the join must not become a
+ // broadcast. Every test below needs both, so they are here rather than in each, and a test states
+ // only what it varies.
+ private def withBloomFilterTables(body: => Unit): Unit = {
+ withTable("bf1", "bf2") {
+ // bf1 is the large application side, 600 rows.
+ spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1")
+ // bf2 is the small creation side, 30 rows, with a selective filter (v = 5 keeps 1 row).
+ spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2")
+ withSQLConf(
+ SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
+ body
+ }
+ }
+ }
+
+ test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too") {
+ // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer
+ // never depends on what the reader managed to do with it.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms >= 1,
+ s"expected >= 1 bloom filter on scan.storageFilters; got $storageBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") {
+ // Nothing is left for phase 2 to prune. The reader would read the same column for the same rows
+ // as a plain scan, since it has to read a key column to evaluate the filter on it, and would
+ // add only the cost of evaluating the predicate outside the generated code.
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5")
+ val plan = df.queryExecution.executedPlan
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters for an all-keys projection; got " +
+ s"$storageBlooms.\nPlan:\n$plan")
+ assert(postScanBlooms >= 1,
+ s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ assert(df.collect().map(_.getLong(0)).toSet == Set(5L),
+ "and the query must still return the joined key")
+ }
+ }
+ }
+
+ test("FileSourceStrategy leaves bloom filter as post-scan FilterExec when conf is off") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, _) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ val postScanBlooms = countBloomFiltersInPostScanFilters(plan)
+ assert(storageBlooms == 0,
+ s"expected no bloom on scan.storageFilters when conf is off; got $storageBlooms")
+ assert(postScanBlooms >= 1,
+ s"expected bloom in post-scan FilterExec when conf is off; got $postScanBlooms.\n" +
+ s"Plan:\n$plan")
+ }
+ }
+ }
+
+ test("the feature still engages when ignoreCorruptFiles is on") {
+ withBloomFilterTables {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.IGNORE_CORRUPT_FILES.key -> "true",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val (plan, rows) = runBloomFilterJoin()
+ val storageBlooms = countBloomFiltersInStorageFilters(plan)
+ assert(storageBlooms == 1,
+ s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan")
+ assert(countBloomFiltersInPostScanFilters(plan) >= 1,
+ s"and the post-scan Filter keeps it.\nPlan:\n$plan")
+ assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSet == Set((5L, 5L)),
+ s"and the query must still return the joined row; got ${rows.mkString(", ")}")
+ // The three assertions above hold whether or not the reader honored the filter, since the
+ // post-scan Filter answers the query either way. The metrics are what say it did.
+ val scan = plan.collect { case s: FileSourceScanExec => s }
+ .find(_.storageFilters.nonEmpty).getOrElse(fail(s"no scan with storage filters:\n$plan"))
+ val excluded =
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP).value +
+ scan.metrics(StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP).value
+ assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded")
+ }
+ }
+ }
+
+ test("a row group over the splice cap is read the plain way, same rows, more bytes") {
+ // Past the cap a row group is read the plain way. Phase 2 takes every projected column, key
+ // columns included, so nothing is buffered. Two things have to hold. The rows must not
+ // change, since a fallback that quietly dropped the predicate would return extra rows. And
+ // the fallback must actually have happened, which is observable. It reads the key column a
+ // second time, so it transfers strictly more bytes. Without that second assertion the test
+ // would pass even if the cap never reached the reader.
+ //
+ // The batch size is not what makes the cap reachable, since the count is examined after
+ // every surviving row. It is here because a row group read the plain way past the cap still
+ // has to emit in batches, and 51 survivors over a capacity of 16 span several of them.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L)))
+
+ def run(maxSplicedBytes: String): (Int, Long) = {
+ withSQLConf(
+ SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key ->
+ maxSplicedBytes,
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") {
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, fileSchema, new StructType(), fileSchema, Nil, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), countingHadoopConf(), metricMap())
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+ readCounting(path, readerFn)
+ }
+ }
+
+ val (splicedRows, splicedBytes) = run("64MB")
+ // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the
+ // row ranges. `k >= 350` keeps a contiguous run, so there is one range to hold.
+ val (plainRows, plainBytes) = run("100b")
+ assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows")
+ assert(plainRows == splicedRows,
+ s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
+ assert(plainBytes > splicedBytes,
+ s"past the cap the key column is read twice, so the read must be larger; " +
+ s"plain=$plainBytes spliced=$splicedBytes")
+ }
+ }
+
+ test("row groups that splice and row groups over the cap, in both orders") {
+ // The emitted batch is one object for the whole read, so its key slots have to follow the path
+ // each row group took. This file makes all four cases occur in order, at 100 rows per row group
+ // and a batch capacity of 16:
+ // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the
+ // 150-byte cap, so the row group splices;
+ // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected
+ // column. The key slots must go back to the persistent vectors here, or the batch reads keys
+ // out of a vector the previous row group's last batch already released;
+ // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving
+ // splicing up released to be allocated afresh;
+ // - rows 301-400: no survivor at all, so the row group is skipped whole.
+ //
+ // Hence the assertion on the values rather than on the row count alone, and the skipped row
+ // group metric, which pins that the file really did split into several row groups.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
+ val m = allMetrics()
+ val k = BoundReference(0, LongType, nullable = true)
+ def between(lo: Long, hi: Long): Expression =
+ And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi)))
+ val filter = createFilter(
+ Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))),
+ StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true))),
+ m,
+ maxSplicedRowGroupBytes = 150L)
+ val (result, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString),
+ capacity = 16)
+ try {
+ val expected = rows.filter { case (key, _) =>
+ key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L)
+ }
+ assert(result == expected,
+ s"expected each surviving key with its own value; got ${result.take(20)}")
+ assert(m.rowGroupsSkipped.value >= 1,
+ s"expected a row group with no survivor at all; got ${m.rowGroupsSkipped.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a row group past the cap charges its second key read against the byte metric") {
+ // Giving splicing up means phase 2 reads the key columns a second time, while the baseline
+ // counts them once, in phase 1. That extra read is a cost against the saving, so the same file
+ // and filter must report less avoided than they do while splicing. Without that term the metric
+ // would credit the feature with bytes it did transfer.
+ withTempDir { dir =>
+ val rows = (1L to 400L).map(i => (i, f"v_$i%04d"))
+ val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L))
+ val fileSchema = StructType(Seq(
+ StructField("k", LongType, nullable = true),
+ StructField("v", StringType, nullable = true)))
+
+ def avoidedBytes(cap: Long, threshold: Long): Long = {
+ val m = allMetrics()
+ val filter = createFilter(
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))),
+ fileSchema, m, cap)
+ val (_, reader) = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => b.column(0).getLong(i), capacity = 16)
+ try m.bytesAvoidedByPageFiltering.value finally reader.close()
+ }
+
+ val spliced = avoidedBytes(64L * 1024 * 1024, 350L)
+ val plain = avoidedBytes(100L, 350L)
+ assert(spliced > 0, s"page filtering has to avoid something here; got $spliced")
+ assert(plain < spliced,
+ s"the second key read must count against the saving; plain=$plain spliced=$spliced")
+
+ // The two halves of the budget are weighed together, and this is the case that says so. Eight
+ // survivors of a long key buffer 8 * (1 + 8) = 72 bytes and fall in one range of 40, so
+ // neither half reaches the 100-byte cap on its own while the sum passes it at the seventh.
+ // Weighing them separately would keep splicing here, and report the larger saving for it.
+ val eightSpliced = avoidedBytes(64L * 1024 * 1024, 393L)
+ val eightPlain = avoidedBytes(100L, 393L)
+ assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced")
+ assert(eightPlain < eightSpliced,
+ s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced")
+ }
+ }
+
+ // Writes a parquet file the low-level way, with no offset index for any column.
+ private def writeParquetFileWithoutOffsetIndex(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]]): String =
+ writeParquetFileByHand(dir, blocks, (_, _) => false)
+
+ // Writes a parquet file the low-level way, with an offset index for the chunks `hasOffsetIndex`
+ // names by block and column. The `ParquetFileWriter.writeDataPage` overloads that take no row
+ // count use parquet's no-op offset index builder, which is how a writer other than parquet-mr's
+ // own produces a chunk that cannot be read in part. Every column is a required int64, so one page
+ // writer serves all.
+ private def writeParquetFileByHand(
+ dir: File,
+ blocks: Seq[Seq[(Long, Long)]],
+ hasOffsetIndex: (Int, Int) => Boolean): String = {
+ val schema = MessageTypeParser.parseMessageType(
+ "message spark_schema { required int64 k; required int64 v; }")
+ val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet")
+ val hadoopPath = new Path(file.getAbsolutePath)
+ val writer = new ParquetFileWriter(
+ HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()),
+ schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8)
+ writer.start()
+ blocks.zipWithIndex.foreach { case (block, blockIdx) =>
+ writer.startBlock(block.size)
+ schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) =>
+ val pageStore = new MemPageStore(block.size)
+ val writeStore = new ColumnWriteStoreV1(pageStore,
+ // The row-count check is what decides when a page is flushed, so without lowering it a
+ // block of a few hundred rows comes out as one page and no column index can narrow it.
+ ParquetProperties.builder().withPageSize(256).withMinRowCountForPageSizeCheck(1)
+ .withDictionaryEncoding(false).build())
+ val columnWriter = writeStore.getColumnWriter(cd)
+ block.foreach { row =>
+ columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0)
+ writeStore.endRecord()
+ }
+ writeStore.flush()
+ writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED)
+ val pageReader = pageStore.getPageReader(cd)
+ var written = 0L
+ while (written < block.size) {
+ val page = pageReader.readPage().asInstanceOf[DataPageV1]
+ if (hasOffsetIndex(blockIdx, colIdx)) {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getValueCount.toLong, page.getRlEncoding,
+ page.getDlEncoding, page.getValueEncoding)
+ } else {
+ writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes,
+ page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding)
+ }
+ written += page.getValueCount
+ }
+ writer.endColumn()
+ }
+ writer.endBlock()
+ }
+ writer.end(new java.util.HashMap[String, String]())
+ file.getAbsolutePath
+ }
+
+ test("a file written with no offset index is read with the filter given up") {
+ // Reading part of a row group needs an offset index, so a file without one cannot be filtered
+ // at page level. What it can still do is skip a row group the filter empties, which needs no
+ // index at all, and that is the split this test pins. The three row groups are ordered so that
+ // the file's first partial read is what discovers the missing index, and the two row groups
+ // after that discovery still get phase 1. The middle one is emptied by the filter and skipped
+ // whole, the last keeps rows and is read whole. The missing index is learned from the read that
+ // fails, so the first row group is the one that discovers it. Spark's own writer always writes
+ // the page index, hence the hand-built file.
+ withTempDir { dir =>
+ val blocks = Seq(
+ (1L to 100L).map(i => (i, i * 10)),
+ (101L to 200L).map(i => (i, i * 10)),
+ (201L to 300L).map(i => (i, i * 10)))
+ val path = writeParquetFileWithoutOffsetIndex(dir, blocks)
+ // The fixture's whole point, asserted rather than assumed.
+ assert(footerChunks(path).forall(_.getOffsetIndexReference == null),
+ "no chunk of the hand-built file may have an offset index")
+ val m = allMetrics()
+ val requested = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", LongType, nullable = false)))
+ // Keeps part of the first row group, none of the second, part of the third.
+ val k = BoundReference(0, LongType, nullable = false)
+ val filter = createFilter(
+ Seq(Or(
+ And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))),
+ And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))),
+ requested,
+ m)
+ // `withLogAppender` returns Unit, so the read's result is captured out of the block.
+ val logAppender = new LogAppender("no offset index")
+ var read: (Seq[(Long, Long)], VectorizedParquetRecordReader) = null
+ withLogAppender(logAppender) {
+ read = readAllWith(path, Seq("k", "v"), filter,
+ (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i)))
+ }
+ val (result, reader) = read
+ try {
+ assert(result == blocks(0) ++ blocks(2),
+ s"the emptied row group is skipped and the other two read whole; got ${result.size} rows")
+ // The footer says which columns have an offset index, so the reader knows before phase 1
+ // buffers anything, and the phase-2 read never asks parquet for a narrowed one. Without
+ // that, the first partially surviving row group copies its survivors and the exception path
+ // throws them away, which is what this WARN reports.
+ assert(!logAppender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ .exists(_.contains("without page-level storage filtering")),
+ "the footer check must get there before the exception path")
+ assert(m.rowGroupsSkipped.value == 1 && m.rowsExcludedByRowGroup.value == 100,
+ s"one row group skipped whole, with its rows counted; got " +
+ s"${m.rowGroupsSkipped.value} and ${m.rowsExcludedByRowGroup.value}")
+ assert(m.rowsExcludedWithinRowGroup.value == 0,
+ s"and nothing excluded inside a row group, which needs the index; got " +
+ s"${m.rowsExcludedWithinRowGroup.value}")
+ } finally {
+ reader.close()
+ }
+ }
+ }
+
+ test("a key column with no offset index does not cost the file its page filtering") {
+ // Reading part of a row group goes through the block's column index store, and parquet builds
+ // that store over every column of the requested schema, emptying it altogether if one of them
+ // has no offset index. So a row group whose key column has none cannot be narrowed either, and
+ // the footer check has to ask about the whole projection. Asking only about the columns phase 2
+ // reads would let such a row group buffer its survivors and then have phase 2 throw, and the
+ // reader answers that by not trusting the page index for the rest of the file. Every row group
+ // after it would then be read whole, which is more than a plain read of the same query, since
+ // that keeps the pushed data filter's page pruning.
+ //
+ // In the file, the first row group's key column has no offset index while its value column
+ // does, and the other two have one for both. The pushed data filter matches one value per row
+ // group, so parquet's statistics filter keeps all three while its column index narrows the two
+ // it can to the page that value is in.
+ withTempDir { dir =>
+ val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+ val path = writeParquetFileByHand(dir, blocks,
+ (blockIdx, colIdx) => blockIdx > 0 || colIdx == 1)
+ // The fixture's whole point, asserted rather than assumed.
+ val indexed = footerChunks(path)
+ .map(c => (c.getPath.toDotString, c.getOffsetIndexReference != null))
+ assert(indexed == Seq(("k", false), ("v", true), ("k", true), ("v", true),
+ ("k", true), ("v", true)),
+ s"the first row group's key column must be the only chunk without one; got $indexed")
+ val schema = StructType(Seq(
+ StructField("k", LongType, nullable = false),
+ StructField("v", LongType, nullable = false)))
+ val pushed = Seq(sources.In("v", Array[Any](1L, 101L, 201L)))
+ // Keeps part of each row group, so a row group that can be narrowed is narrowed.
+ val storageFilters =
+ Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(50L)))
+ val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters(
+ spark, schema, new StructType(), schema, pushed, storageFilters,
+ Map(FileFormat.OPTION_RETURNING_BATCH -> "true"),
+ spark.sessionState.newHadoopConf(), metricMap())
+ .getOrElse(fail("ParquetFileFormat must answer with a reader"))
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new File(path).length())
+ val logAppender = new LogAppender("no offset index")
+ var values: Seq[Long] = null
+ withLogAppender(logAppender) {
+ values = readerFn(file).asInstanceOf[Iterator[Object]].flatMap {
Review Comment:
Fixed. A `drain` helper consumes the iterator and closes it in a `finally`, and all three sites use it.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled once the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Set to true once this reader has no rows left to return, which is when the row groups run out
+ * or when the filter rejects the whole file.
+ */
+ private boolean hitEndOfData = false;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch is a view whose slots alias these vectors (non-key and partition) and the head of
+ // each survivor queue (key slots), so closing it would re-close a shared vector and free the
+ // same buffer twice. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ @Override
+ public float getProgress() {
+ // Under a storage filter, rowsReturned counts survivors while totalRowCount is the count before
+ // the filter narrowed anything, so the ratio neither reaches 1 on its own nor stays below it. A
+ // row group read whole after the page index is given up can return more rows than that count.
+ // `hitEndOfData` is the real terminator here, and the ratio is clamped to what it means.
+ if (hitEndOfData) return 1.0f;
+ return Math.min(1.0f, super.getProgress());
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file. True, or an error, reads it the way a plain scan would, under the same
+ * fail-open rule as a row's value, since the constant can be one the predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (RuntimeException e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ hitEndOfData = true;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Whether this read has returned every row it is going to.
+ *
+ * <p>{@code totalRowCount} is how many rows the file yields before this filter narrows anything,
+ * so it is a terminator only while there is no filter. Under one it is an upper bound the read
+ * never reaches, and a row group read whole after the page index was given up can pass it. What
+ * ends such a read is {@link #hitEndOfData}, set when the row groups run out or when the filter
+ * rejects the whole file.
+ */
+ @Override
+ protected boolean noMoreRows() {
+ return hitEndOfData || (storageFilter == null && super.noMoreRows());
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
Review Comment:
A correction to my reply above. Deleting the override tripped MiMa, as a `DirectAbstractMethodProblem` reported against `org.apache.hadoop.mapreduce.RecordReader.initialize`, which the `sql.execution.*` exclude does not cover. So the override is back and only calls `super.initialize`. The double initialization is gone either way, and the test still covers it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4136316102
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled once the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Set to true once this reader has no rows left to return, which is when the row groups run out
+ * or when the filter rejects the whole file.
+ */
+ private boolean hitEndOfData = false;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch is a view whose slots alias these vectors (non-key and partition) and the head of
+ // each survivor queue (key slots), so closing it would re-close a shared vector and free the
+ // same buffer twice. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ @Override
+ public float getProgress() {
+ // Under a storage filter, rowsReturned counts survivors while totalRowCount is the count before
+ // the filter narrowed anything, so the ratio neither reaches 1 on its own nor stays below it. A
+ // row group read whole after the page index is given up can return more rows than that count.
+ // `hitEndOfData` is the real terminator here, and the ratio is clamped to what it means.
+ if (hitEndOfData) return 1.0f;
+ return Math.min(1.0f, super.getProgress());
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file. True, or an error, reads it the way a plain scan would, under the same
+ * fail-open rule as a row's value, since the constant can be one the predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (RuntimeException e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ hitEndOfData = true;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Whether this read has returned every row it is going to.
+ *
+ * <p>{@code totalRowCount} is how many rows the file yields before this filter narrows anything,
+ * so it is a terminator only while there is no filter. Under one it is an upper bound the read
+ * never reaches, and a row group read whole after the page index was given up can pass it. What
+ * ends such a read is {@link #hitEndOfData}, set when the row groups run out or when the filter
+ * rejects the whole file.
+ */
+ @Override
+ protected boolean noMoreRows() {
+ return hitEndOfData || (storageFilter == null && super.noMoreRows());
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Should parquet ever say
+ // otherwise, the filter is given up for this row group, which reads what a plain read
+ // reads. The alternative is to fail a query a plain read answers.
+ abandonSplicing();
+ survivors = null;
+ } else {
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ if (baselineRows != blockRowCount) {
+ // The retry avoids the same wall because a store missing one column's offset index
+ // reports no column index either, so these ranges cover the whole block and parquet reads
+ // it without consulting an index. That is three parquet internals deep, so it is not
+ // assumed, and if they are narrower the block is read whole instead, which is a superset
+ // of them. The extra rows cost the post-scan Filter work, where failing here would fail a
+ // query a plain read answers, and under `ignoreCorruptFiles` would be read as a corrupt
+ // file and drop the rest of this one.
+ finalRanges = RowRanges.createSingle(blockRowCount);
+ finalRowCount = blockRowCount;
+ }
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ long keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up, since it read what a plain
+ // scan reads and the saving is a certain zero.
+ if (!filterGivenUp) {
+ // The same ranges as the baseline when nothing was pruned, and then the walk would arrive
+ // at the same number a second time.
+ long phase2Bytes = finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += blockAccounting.compressedBytesForRowRanges(blockIdx,
+ keyOnlyPaths, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ long filteredRows = baselineRows - keptRows;
+ if (filteredRows > 0) m.rowsExcludedWithinRowGroup().add(filteredRows);
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ RowRanges readerRanges = readerRangesFor(finalRanges, finalRowCount, blockRowCount);
+ installRowGroup(dataPages, readerRanges, keptRows);
+ return true;
+ }
+ hitEndOfData = true;
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queues at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ return spliceCurrentRowGroup && isKeyTopLevel[slot];
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses, since the rows its column index already excluded were never this filter's
+ // to save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ *
+ * <p>A file this reader can no longer read in part still comes through here, since these are the
+ * rows a plain read of the same query would read, and leaving them to the post-scan Filter
+ * instead would make the scan slower than having no storage filter at all.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * What the surviving rows of the current row group cost to hold as row ranges, which is the half
+ * of the budget that a row group buffering nothing still pays. A filter whose survivors are
+ * scattered makes one range per surviving row, and phase 2 needs the whole set to select its
+ * pages. One set is held, not one per column reader, and the readers of a row group walk that
+ * same list, each with a cursor of its own.
+ */
+ private static long rowRangeStateBytes(long rangeCount) {
+ // Parquet's own `RowRanges.Range` holds two longs, plus its object header and a list slot.
+ return rangeCount * 40L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group. The
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ ensureKeyScratchAllocated();
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below, which is the bytes buffered
+ // for splicing and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
+ if (!canNarrowRowGroup) {
+ // Nothing narrower than the whole row group can be read, so whether anything survives
+ // at all was the only thing left to learn, and it does. Stop here, since the ranges are
+ // not needed and the rest of the key column need not be decoded or evaluated. The
+ // row groups the filter empties are still skipped, which is the case that returns a
+ // non-null empty range set below.
+ abandonSplicing();
+ return null;
+ }
+ finalRangesBuilder.addSelectedRow(blockRow);
+ if (blockRow != previousSurvivor + 1) survivorRangeCount++;
+ previousSurvivor = blockRow;
+ if (spliceCurrentRowGroup) splicedBytes += appendSurvivorRowToAccumulators(r);
+ // Both halves of what this row group retains grow per survivor, and either can cross
+ // the budget on its own, so they are weighed together here and nowhere else. The cheaper
+ // concession comes first, which is to release the buffer, and the filter goes too if the
+ // ranges alone still do not fit.
+ long rangeBytes = rowRangeStateBytes(survivorRangeCount);
+ if (splicedBytes + rangeBytes > cap) {
+ // The cheaper concession first, releasing the buffer, which is a no-op for a row group
+ // that has already given splicing up.
+ abandonSplicing();
+ splicedBytes = 0L;
+ if (rangeBytes > cap) {
+ // The ranges being built are about to be thrown away, so stop evaluating the rest.
+ return null;
+ }
+ }
+ } catch (RuntimeException e) {
Review Comment:
Fixed. Both catches take `Exception` now, and `logFilterGivenUpOnError` does too. Nothing inside either `try` declares a checked exception, so this widens what is caught and nothing else.
The test you suggest is in. It uses a test expression that throws a checked `SparkException` on one key value, the way a Hive UDF key does through `FAILED_EXECUTE_UDF`. It runs twice, once per row and once as the constant of a file missing its key column, since that is the second catch. Narrowing either catch back to `RuntimeException` fails its test.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled once the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Set to true once this reader has no rows left to return, which is when the row groups run out
+ * or when the filter rejects the whole file.
+ */
+ private boolean hitEndOfData = false;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch is a view whose slots alias these vectors (non-key and partition) and the head of
+ // each survivor queue (key slots), so closing it would re-close a shared vector and free the
+ // same buffer twice. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ @Override
+ public float getProgress() {
+ // Under a storage filter, rowsReturned counts survivors while totalRowCount is the count before
+ // the filter narrowed anything, so the ratio neither reaches 1 on its own nor stays below it. A
+ // row group read whole after the page index is given up can return more rows than that count.
+ // `hitEndOfData` is the real terminator here, and the ratio is clamped to what it means.
+ if (hitEndOfData) return 1.0f;
+ return Math.min(1.0f, super.getProgress());
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file. True, or an error, reads it the way a plain scan would, under the same
+ * fail-open rule as a row's value, since the constant can be one the predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (RuntimeException e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ hitEndOfData = true;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Whether this read has returned every row it is going to.
+ *
+ * <p>{@code totalRowCount} is how many rows the file yields before this filter narrows anything,
+ * so it is a terminator only while there is no filter. Under one it is an upper bound the read
+ * never reaches, and a row group read whole after the page index was given up can pass it. What
+ * ends such a read is {@link #hitEndOfData}, set when the row groups run out or when the filter
+ * rejects the whole file.
+ */
+ @Override
+ protected boolean noMoreRows() {
+ return hitEndOfData || (storageFilter == null && super.noMoreRows());
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
Review Comment:
Fixed the second way, at the source. `VectorizedParquetRecordReader` no longer overrides the 2-arg `initialize`, since super's already delegates to the 5-arg overload, which runs `initializeInternal()`. So the plain reader stops initializing twice as well.
A new test initializes the reader through the 2-arg overload on an all-keys projection, which is the decline you describe. With the old override put back it fails with exactly the NPE you predicted.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java:
##########
@@ -83,54 +79,45 @@ final class ParquetReadState {
* levels. */
boolean shouldSkip;
- ParquetReadState(
- ColumnDescriptor descriptor,
- boolean isRequired,
- PrimitiveIterator.OfLong rowIndexes) {
+ private ParquetReadState(ColumnDescriptor descriptor, boolean isRequired) {
this.maxRepetitionLevel = descriptor.getMaxRepetitionLevel();
this.maxDefinitionLevel = descriptor.getMaxDefinitionLevel();
this.isRequired = isRequired;
- this.rowRanges = constructRanges(rowIndexes);
- nextRange();
}
/**
- * Construct a list of row ranges from the given `rowIndexes`. For example, suppose the
- * `rowIndexes` are `[0, 1, 2, 4, 5, 7, 8, 9]`, it will be converted into 3 row ranges:
- * `[0-2], [4-5], [7-9]`.
+ * The state for reading {@code descriptor}, told which rows to include the best way the caller
+ * can say it.
+ *
+ * <p>{@code rowRanges} is the rows the read was asked for, when whoever asked knew them. A caller
+ * that filtered the rows itself holds their ranges, and every column reader of the row group can
+ * then walk that one list. Null says it does not, and the page store is left to describe its
+ * rows, which it can only do as {@code rowIndexes}, one index per row. No row indexes in turn
+ * means every row of the chunk is included.
*/
- private Iterator<RowRange> constructRanges(PrimitiveIterator.OfLong rowIndexes) {
- if (rowIndexes == null) {
- return null;
- }
-
- List<RowRange> rowRanges = new ArrayList<>();
- long currentStart = Long.MIN_VALUE;
- long previous = Long.MIN_VALUE;
-
- while (rowIndexes.hasNext()) {
- long idx = rowIndexes.nextLong();
- if (currentStart == Long.MIN_VALUE) {
- currentStart = idx;
- } else if (previous + 1 != idx) {
- RowRange range = new RowRange(currentStart, previous);
- rowRanges.add(range);
- currentStart = idx;
- }
- previous = idx;
- }
-
- if (previous != Long.MIN_VALUE) {
- rowRanges.add(new RowRange(currentStart, previous));
+ static ParquetReadState forRead(
+ ColumnDescriptor descriptor,
+ boolean isRequired,
+ RowRanges rowRanges,
+ PrimitiveIterator.OfLong rowIndexes) {
+ ParquetReadState state;
+ if (rowRanges != null) {
Review Comment:
Added, in `ParquetReadState.forRead`, since that is where both arrive. Ranges with no row indexes throw an `IllegalStateException` that names the column. That covers phase 1 and phase 2 alike, and any other caller that pairs ranges with a store from `readRowGroup`.
The test bridge used to allow ranges alone, and `VectorizedRleValuesReaderSuite` relied on that. Its range arm now passes both, as production does, and a new test there asserts that ranges alone are refused.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetBlockAccounting.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.column.columnindex.OffsetIndex;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+
+/**
+ * What a partial read of a Parquet file transfers, and whether one is possible at all, answered
+ * from the footer the {@link ParquetFileReader} already holds. Every method here is metadata
+ * arithmetic and causes no IO of its own, which is what lets a reader report these numbers per row
+ * group.
+ *
+ * <p>One instance per file, because the warning below is reported once per file.
+ */
+final class ParquetBlockAccounting {
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(ParquetBlockAccounting.class);
+
+ private final ParquetFileReader reader;
+
+ /** Whether the byte count has already been reported as undercounting on this file. */
+ private boolean loggedMissingStoreEntry;
+
+ ParquetBlockAccounting(ParquetFileReader reader) {
+ this.reader = reader;
+ }
+
+ /** Resolves a column list's leaf paths, so the per-row-group footer walks do not have to. */
+ static Set<ColumnPath> pathsOf(List<ColumnDescriptor> columns) {
+ Set<ColumnPath> paths = new HashSet<>();
+ for (ColumnDescriptor column : columns) {
+ paths.add(ColumnPath.get(column.getPath()));
+ }
+ return paths;
+ }
+
+ /**
+ * Whether every one of {@code paths} that this file has can be read in part, which needs an
+ * offset index for it in this block. A column the file does not have is no obstacle, since a read
+ * never reads it and it is not in the walk to begin with.
+ */
+ boolean hasOffsetIndexes(int blockIndex, Set<ColumnPath> paths) {
+ for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) {
+ if (paths.contains(chunk.getPath()) && chunk.getOffsetIndexReference() == null) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Compressed bytes a reader transfers for the leaf columns named by {@code paths} when it reads
+ * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are
+ * included, since both are read whenever any page of a chunk is read. {@code rowRangeCount} is
+ * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it.
+ *
+ * <p>Two sources, chosen so this never causes IO of its own:
+ * <ul>
+ * <li>{@code rowRanges} covers the whole block, and the answer is the sum of the chunks'
+ * {@code getTotalSize()}, which is already in the footer. This is the case that matters.
+ * Whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges}
+ * is necessarily the whole block, because a narrower range can only come from column-index
+ * filtering, which builds the store as a side effect.
+ * <li>{@code rowRanges} is a strict subset, and the walk goes over the offset index, as
+ * parquet's own read path does, adding the dictionary page the way
+ * {@code calculateOffsetRanges} does. The store is guaranteed to exist here, so the walk is
+ * pure metadata arithmetic. For ranges a reader narrowed itself, which column-index
+ * filtering had no hand in, that guarantee is an ordering one, since the reader's own read
+ * of those ranges built the store first.
+ * </ul>
+ *
+ * <p>Columns absent from this physical file (schema evolution) contribute nothing, which is
+ * correct, since the reader transfers nothing for them. They fall out of the walk by themselves,
+ * which goes over the block's chunks rather than over {@code paths}.
+ */
+ long compressedBytesForRowRanges(
+ int blockIndex,
+ Set<ColumnPath> paths,
+ RowRanges rowRanges,
+ long rowRangeCount) {
+ if (paths.isEmpty() || rowRangeCount == 0) {
+ return 0L;
+ }
+ BlockMetaData block = reader.getRowGroups().get(blockIndex);
+ long blockRowCount = block.getRowCount();
+ boolean wholeBlock = rowRangeCount == blockRowCount;
+ ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex);
+ long total = 0L;
+ for (ColumnChunkMetaData chunk : block.getColumns()) {
+ ColumnPath path = chunk.getPath();
+ if (!paths.contains(path)) {
+ // A leaf of the file this walk is not about, either not projected at all, or projected but
+ // read in another phase than the one being measured.
+ continue;
+ }
+ if (wholeBlock) {
+ total += chunk.getTotalSize();
+ continue;
+ }
+ OffsetIndex offsetIndex;
+ try {
+ offsetIndex = ciStore.getOffsetIndex(path);
+ } catch (MissingOffsetIndexException e) {
+ continue;
+ }
+ if (offsetIndex == null) {
+ // The store answers null, rather than throwing, for a path it was not built with. That can
+ // only mean the block's store was built while a narrower schema was requested than this
+ // walk asks about, an ordering bug rather than a property of the file, and the footer tells
+ // the two apart. It is reported rather than thrown, because this walk only produces a
+ // counter, and `ignoreCorruptFiles` turns any exception from a reader into a silently
+ // truncated file, so a byte metric must not be able to change the answer.
+ if (chunk.getOffsetIndexReference() != null && !loggedMissingStoreEntry) {
+ loggedMissingStoreEntry = true;
+ LOG.warn("Undercounting the storage filter's avoided bytes for {}: column {} of row "
+ + "group {} has an offset index the block's column index store was not built with",
+ MDC.of(LogKeys.PATH, reader.getFile()),
+ MDC.of(LogKeys.COLUMN_NAME, path.toDotString()),
+ MDC.of(LogKeys.INDEX, blockIndex));
+ }
+ continue;
+ }
+ // The dictionary page is read whenever any data page of the chunk is, so count it here the
+ // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does.
+ total += dictionaryPageSize(chunk);
+ int pageCount = offsetIndex.getPageCount();
+ for (int i = 0; i < pageCount; i++) {
+ long from = offsetIndex.getFirstRowIndex(i);
+ long to = offsetIndex.getLastRowIndex(i, blockRowCount);
+ if (rowRanges.isOverlapping(from, to)) {
+ total += offsetIndex.getCompressedPageSize(i);
+ }
+ }
+ }
+ return total;
+ }
+
+ /**
+ * Compressed size of a chunk's dictionary page, or 0 if it has none.
+ * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset
+ * when there is a valid one, so the gap up to the first data page is exactly the dictionary page.
+ */
+ private static long dictionaryPageSize(ColumnChunkMetaData chunk) {
+ long startingPos = chunk.getStartingPos();
+ long firstDataPageOffset = chunk.getFirstDataPageOffset();
+ return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L;
Review Comment:
Fixed as you suggest. The upper bound is `offsetIndex.getOffset(0)` now, guarded by `getPageCount() > 0`, which is what `calculateOffsetRanges` is handed. I checked PARQUET-1977 in parquet-java. Before 1.12.0 the writer took the chunk's first data page offset at `startColumn`, which is before the dictionary page, so the gap read as 0 for exactly the files you name.
There is no test for it, since that needs a file written by parquet-mr 1.11. For a file a current writer produces the two bounds are equal, and the existing byte tests pass unchanged.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled once the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Set to true once this reader has no rows left to return, which is when the row groups run out
+ * or when the filter rejects the whole file.
+ */
+ private boolean hitEndOfData = false;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch is a view whose slots alias these vectors (non-key and partition) and the head of
+ // each survivor queue (key slots), so closing it would re-close a shared vector and free the
+ // same buffer twice. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ @Override
+ public float getProgress() {
+ // Under a storage filter, rowsReturned counts survivors while totalRowCount is the count before
+ // the filter narrowed anything, so the ratio neither reaches 1 on its own nor stays below it. A
+ // row group read whole after the page index is given up can return more rows than that count.
+ // `hitEndOfData` is the real terminator here, and the ratio is clamped to what it means.
+ if (hitEndOfData) return 1.0f;
+ return Math.min(1.0f, super.getProgress());
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file. True, or an error, reads it the way a plain scan would, under the same
+ * fail-open rule as a row's value, since the constant can be one the predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (RuntimeException e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ hitEndOfData = true;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Whether this read has returned every row it is going to.
+ *
+ * <p>{@code totalRowCount} is how many rows the file yields before this filter narrows anything,
+ * so it is a terminator only while there is no filter. Under one it is an upper bound the read
+ * never reaches, and a row group read whole after the page index was given up can pass it. What
+ * ends such a read is {@link #hitEndOfData}, set when the row groups run out or when the filter
+ * rejects the whole file.
+ */
+ @Override
+ protected boolean noMoreRows() {
+ return hitEndOfData || (storageFilter == null && super.noMoreRows());
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
Review Comment:
Both sites assert now. The give-up could only meet the same `null` again in phase 2, since neither condition depends on the columns requested. It was an `IllegalStateException` originally, and one of my own cleanup rounds turned it into the give-up without noticing that.
The empty-block comment names the real trigger now, this reader's own `RowRanges.createSingle(0)` under `pageIndexDisabled`.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -752,15 +774,36 @@ case class FileSourceScanExec(
lazy val inputRDD: RDD[InternalRow] = {
val options = relation.options +
(FileFormat.OPTION_RETURNING_BATCH -> supportsColumnar.toString)
- val readFile: (PartitionedFile) => Iterator[InternalRow] =
- relation.fileFormat.buildReaderWithPartitionValues(
+ val hadoopConf = getHadoopConf(relation.sparkSession, relation.options)
Review Comment:
Fixed. Each builder gets its own `getHadoopConf(...)` again, so nothing a declining builder sets up can reach the fallback. The second copy is made only when a format declines.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled once the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Set to true once this reader has no rows left to return, which is when the row groups run out
+ * or when the filter rejects the whole file.
+ */
+ private boolean hitEndOfData = false;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch is a view whose slots alias these vectors (non-key and partition) and the head of
+ // each survivor queue (key slots), so closing it would re-close a shared vector and free the
+ // same buffer twice. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ @Override
+ public float getProgress() {
+ // Under a storage filter, rowsReturned counts survivors while totalRowCount is the count before
+ // the filter narrowed anything, so the ratio neither reaches 1 on its own nor stays below it. A
+ // row group read whole after the page index is given up can return more rows than that count.
+ // `hitEndOfData` is the real terminator here, and the ratio is clamped to what it means.
+ if (hitEndOfData) return 1.0f;
+ return Math.min(1.0f, super.getProgress());
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file. True, or an error, reads it the way a plain scan would, under the same
+ * fail-open rule as a row's value, since the constant can be one the predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (RuntimeException e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ hitEndOfData = true;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Whether this read has returned every row it is going to.
+ *
+ * <p>{@code totalRowCount} is how many rows the file yields before this filter narrows anything,
+ * so it is a terminator only while there is no filter. Under one it is an upper bound the read
+ * never reaches, and a row group read whole after the page index was given up can pass it. What
+ * ends such a read is {@link #hitEndOfData}, set when the row groups run out or when the filter
+ * rejects the whole file.
+ */
+ @Override
+ protected boolean noMoreRows() {
+ return hitEndOfData || (storageFilter == null && super.noMoreRows());
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Should parquet ever say
+ // otherwise, the filter is given up for this row group, which reads what a plain read
+ // reads. The alternative is to fail a query a plain read answers.
+ abandonSplicing();
+ survivors = null;
+ } else {
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ if (baselineRows != blockRowCount) {
+ // The retry avoids the same wall because a store missing one column's offset index
+ // reports no column index either, so these ranges cover the whole block and parquet reads
+ // it without consulting an index. That is three parquet internals deep, so it is not
+ // assumed, and if they are narrower the block is read whole instead, which is a superset
+ // of them. The extra rows cost the post-scan Filter work, where failing here would fail a
+ // query a plain read answers, and under `ignoreCorruptFiles` would be read as a corrupt
+ // file and drop the rest of this one.
+ finalRanges = RowRanges.createSingle(blockRowCount);
+ finalRowCount = blockRowCount;
+ }
+ fileReader.setRequestedSchema(requestedColumns);
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ }
+ if (dataPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, both excluded above. Match phase 1 and fail with a message rather than an NPE.
+ throw new IllegalStateException(
+ "No data pages for row group " + blockIdx + " despite " + finalRowCount
+ + " rows to read");
+ }
+ long keptRows = dataPages.getRowCount();
+ // Nothing is reported for a row group whose filter was given up, since it read what a plain
+ // scan reads and the saving is a certain zero.
+ if (!filterGivenUp) {
+ // The same ranges as the baseline when nothing was pruned, and then the walk would arrive
+ // at the same number a second time.
+ long phase2Bytes = finalRowCount == baselineRows
+ ? nonKeyBaselineBytes
+ : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, finalRanges, finalRowCount);
+ if (!spliceCurrentRowGroup) {
+ // This row group gave splicing up, so phase 2 read the key columns a second time. The
+ // baseline counts them once, in phase 1, so the extra read is a cost against it.
+ phase2Bytes += blockAccounting.compressedBytesForRowRanges(blockIdx,
+ keyOnlyPaths, finalRanges, finalRowCount);
+ }
+ // `SQLMetric.add` ignores a negative value, so a row group that read more than the
+ // baseline after giving splicing up contributes nothing rather than subtracting.
+ m.bytesAvoidedByPageFiltering().add(nonKeyBaselineBytes - phase2Bytes);
+ }
+ long filteredRows = baselineRows - keptRows;
+ if (filteredRows > 0) m.rowsExcludedWithinRowGroup().add(filteredRows);
+
+ // Phase 2 read exactly these rows, so its readers are handed the ranges rather than left to
+ // rebuild them from the store's one row index per row.
+ RowRanges readerRanges = readerRangesFor(finalRanges, finalRowCount, blockRowCount);
+ installRowGroup(dataPages, readerRanges, keptRows);
+ return true;
+ }
+ hitEndOfData = true;
+ return false;
+ }
+
+ /**
+ * A spliced row group's key slots take their values from the survivor queues at emit, so phase 2
+ * reads no pages for them. Key columns are primitive, which is what the base class asks of a slot
+ * answered true here.
+ */
+ @Override
+ protected boolean suppliesOwnVector(int slot) {
+ return spliceCurrentRowGroup && isKeyTopLevel[slot];
+ }
+
+ /**
+ * Counts a file the filter rejects whole, which happens when every key column is missing from it
+ * and the predicate is constant-false for the value the reader would have materialized. Every row
+ * group counts as skipped and every projected byte as avoided, which is what the counters mean
+ * for a row group the filter empties.
+ */
+ private void recordFileSkipped() {
+ StorageFilterMetrics m = storageFilter.metrics();
+ for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) {
+ // Measured against the rows the pushed data filter kept, which is the baseline every other
+ // skip path uses, since the rows its column index already excluded were never this filter's
+ // to save. Resolving them again is a cache hit whenever the two can differ, because
+ // `getFilteredRecordCount()` at initialize resolved every block's ranges then.
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) continue;
+ RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ long survivingRows = blockRanges.rowCount();
+ if (survivingRows == 0) continue;
+ // The key columns are missing from this file, so they contribute nothing to the walk, and the
+ // whole projection is what a plain read would have transferred.
+ long avoidedBytes = blockAccounting.compressedBytesForRowRanges(
+ blockIdx, requestedPaths, blockRanges, survivingRows);
+ m.recordRowGroupSkipped(survivingRows, avoidedBytes);
+ }
+ }
+
+ /**
+ * The rows of a block the pushed data filter allows, at column-index granularity, or every row of
+ * the block when {@link #pageIndexDisabled} says the page index must not be used.
+ *
+ * <p>{@code getRowRanges} asks only whether a filter is pushed, not whether column-index
+ * filtering is still enabled, so this is where that conf is honoured. Every phase reads within
+ * these ranges, so narrowing them by an index the read may not trust would cost rows.
+ *
+ * <p>A file this reader can no longer read in part still comes through here, since these are the
+ * rows a plain read of the same query would read, and leaving them to the post-scan Filter
+ * instead would make the scan slower than having no storage filter at all.
+ */
+ private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) {
+ return pageIndexDisabled
+ ? RowRanges.createSingle(blockRowCount)
+ : fileReader.getRowRanges(blockIdx);
+ }
+
+ /**
+ * The ranges to hand the column readers of a read that asked for {@code rowRanges}, which select
+ * {@code rowCount} of the block's {@code blockRowCount} rows. They then find those rows in the
+ * pages with no per-row work.
+ *
+ * <p>None for a read of every row, since parquet reads the block whole in that case, and a reader
+ * that filters by nothing is cheaper than one that filters by everything.
+ */
+ private static RowRanges readerRangesFor(
+ RowRanges rowRanges, long rowCount, long blockRowCount) {
+ return rowCount == blockRowCount ? null : rowRanges;
+ }
+
+ /**
+ * What the surviving rows of the current row group cost to hold as row ranges, which is the half
+ * of the budget that a row group buffering nothing still pays. A filter whose survivors are
+ * scattered makes one range per surviving row, and phase 2 needs the whole set to select its
+ * pages. One set is held, not one per column reader, and the readers of a row group walk that
+ * same list, each with a cursor of its own.
+ */
+ private static long rowRangeStateBytes(long rangeCount) {
+ // Parquet's own `RowRanges.Range` holds two longs, plus its object header and a list slot.
+ return rangeCount * 40L;
+ }
+
+ /**
+ * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in
+ * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row
+ * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never
+ * read.
+ *
+ * <p>Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path
+ * to splice, until the buffer passes its cap. From there the row group is evaluated without
+ * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns
+ * again along with everything else.
+ *
+ * <p>Returns null once the budget makes the reader give the filter up for this row group. The
+ * ranges built so far are then incomplete, and the caller reads the row group the plain way.
+ */
+ private RowRanges evaluateStorageFilter(
+ PageReadStore keyPages,
+ RowRanges pushedFilterRanges,
+ long pushedFilterRowCount,
+ long blockRowCount,
+ boolean canNarrowRowGroup) throws IOException {
+ ensureKeyScratchAllocated();
+ long remaining = pushedFilterRowCount;
+ // Phase 1 read exactly the pushed filter's rows, so its readers are handed those ranges.
+ RowRanges keyReaderRanges =
+ readerRangesFor(pushedFilterRanges, pushedFilterRowCount, blockRowCount);
+ VectorizedColumnReader[] readers = new VectorizedColumnReader[keyColumns.length];
+ for (int i = 0; i < readers.length; i++) {
+ KeyColumn key = keyColumns[i];
+ readers[i] =
+ newColumnReader(key.descriptor(), key.required(), keyPages, keyReaderRanges);
+ }
+ if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated();
+
+ PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
+ RowRanges.Builder finalRangesBuilder = RowRanges.builder();
+ // What this row group retains, weighed against the budget below, which is the bytes buffered
+ // for splicing and the ranges the surviving rows fall into.
+ long splicedBytes = 0L;
+ long survivorRangeCount = 0L;
+ long previousSurvivor = -2L;
+ long cap = storageFilter.maxSplicedRowGroupBytes();
+ // Out of the loop, because reaching it through the filter resolves a `lazy val`, which is a
+ // volatile read the loop would pay per row.
+ BasePredicate predicate = storageFilter.preparedPredicate();
+ while (remaining > 0) {
+ int num = (int) Math.min((long) capacity, remaining);
+ for (int i = 0; i < keyScratchVectors.length; i++) {
+ keyScratchVectors[i].reset();
+ readers[i].readBatch(num, keyScratchVectors[i], null, null);
+ }
+ keyScratchBatch.setNumRows(num);
+ for (int r = 0; r < num; r++) {
+ long blockRow = rowIndexIter.nextLong();
+ try {
+ if (!predicate.eval(keyScratchBatch.getRow(r))) continue;
Review Comment:
Added to the description's follow-up list, next to the vector-evaluation item and with your distinction between the two. I kept it out of this PR because it is a second evaluation path in phase 1's hot loop, with cases of its own to test. Those are a chunk that falls back to plain encoding part way, nulls, and an id that throws. It also saves CPU rather than IO, and IO is what this PR is about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58895:
URL: https://github.com/apache/spark/pull/58895#discussion_r4136317873
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import org.apache.spark.SparkContext
+import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Predicate}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types.{DataType, StructType}
+
+/**
+ * The SQL metrics the reader updates while applying a [[ParquetStorageFilter]], created by
+ * `ParquetFileFormat.storageFilterMetrics`. Every counter is scoped to what the storage filter
+ * added on top of a read of the same projection without one.
+ *
+ * - [[rowGroupsSkipped]] counts row groups whose data columns were never read.
+ * - [[rowsExcludedByRowGroup]] sums those skipped row groups' rows, counting per block the rows
+ * that survived the pushed data filter.
+ * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept.
+ * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this
+ * projection would have transferred for the rows that survived the pushed data filter. Phase 1
+ * reads the key columns of every block, so key bytes are never part of it, and it is zero on an
+ * all-keys projection, which can avoid nothing.
+ * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the
+ * bytes phase 2 read, which is what `finalRanges` page selection pruned.
+ *
+ * The row counters' suffix says where a row was excluded, not whether reading it was avoided. A row
+ * excluded inside a kept row group may have been read as part of a page that held a survivor, or
+ * not read at all because phase 2 skipped its page. An all-keys projection has no page filtering at
+ * all, and [[rowsExcludedWithinRowGroup]] still counts every row the filter dropped.
+ */
+case class StorageFilterMetrics(
+ rowGroupsSkipped: SQLMetric,
+ rowsExcludedByRowGroup: SQLMetric,
+ rowsExcludedWithinRowGroup: SQLMetric,
+ bytesAvoidedByRowGroup: SQLMetric,
+ bytesAvoidedByPageFiltering: SQLMetric) {
+
+ /** Counts a row group whose data columns the filter kept the reader from touching at all. */
+ def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
+ rowGroupsSkipped.add(1L)
+ rowsExcludedByRowGroup.add(excludedRows)
+ bytesAvoidedByRowGroup.add(avoidedBytes)
+ }
+
+ /** These counters keyed for the scan, which carries them without naming any of them. */
+ def toMap: Map[String, SQLMetric] = Map(
+ StorageFilterMetrics.ROW_GROUPS_SKIPPED -> rowGroupsSkipped,
+ StorageFilterMetrics.ROWS_EXCLUDED_BY_ROW_GROUP -> rowsExcludedByRowGroup,
+ StorageFilterMetrics.ROWS_EXCLUDED_WITHIN_ROW_GROUP -> rowsExcludedWithinRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_ROW_GROUP -> bytesAvoidedByRowGroup,
+ StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING -> bytesAvoidedByPageFiltering)
+}
+
+object StorageFilterMetrics {
+
+ // The keys these counters are exposed under, next to the fields they name. What they count is
+ // this reader's vocabulary, which is why the format creates them rather than the scan.
+ val ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped"
+ val ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup"
+ val ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup"
+ val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
+ val BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering"
+
+ /** A fresh set of counters, with the labels the SQL UI shows. */
+ def create(sparkContext: SparkContext): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped =
+ SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"),
+ rowsExcludedByRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"),
+ rowsExcludedWithinRowGroup =
+ SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"),
+ bytesAvoidedByRowGroup = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (whole row group)"),
+ bytesAvoidedByPageFiltering = SQLMetrics.createSizeMetric(
+ sparkContext, "bytes avoided by storage filter (page filtering)"))
+
+ /**
+ * The counters the scan carries back, which are the ones [[create]] made: the map is this
+ * format's own transport, and a scan with storage filters on a Parquet relation has no other.
+ */
+ def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = StorageFilterMetrics(
+ rowGroupsSkipped = metrics(ROW_GROUPS_SKIPPED),
+ rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
+ rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
+ bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+}
+
+/**
+ * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read
+ * key-column pages first, evaluate this filter per row to decide which rows survive, and skip
+ * data-column pages that do not overlap any surviving row range.
+ *
+ * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf
+ * columns referenced by the filter. [[preparedPredicate]] expects a row whose fields are those key
+ * columns in that order. The predicate's references are rewritten to [[BoundReference]]s pointing
+ * at positions 0..(keyColumnIndices.length - 1) when the filter is built.
+ */
+class ParquetStorageFilter private (
+ val keyColumnIndices: Array[Int],
+ private var boundExpression: Expression,
+ val metrics: StorageFilterMetrics,
+ val maxSplicedRowGroupBytes: Long) extends Serializable {
+
+ // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer
+ // construction to first use on the executor.
+ @transient private lazy val predicate: BasePredicate = {
+ // The field below is nulled the first time this runs, and the copy that reaches every task is
+ // the driver's. So a driver-side evaluation would leave each executor building a predicate from
+ // nothing, which is named here rather than left to an NPE from `Predicate.create`.
+ require(boundExpression != null,
+ "a storage filter must only be evaluated on an executor, and this copy has released the " +
+ "expression it would build its predicate from")
+ val created = Predicate.create(boundExpression)
+ // A prepared bloom reaches the executor as a binary Literal inside this expression, up to
+ // `spark.sql.optimizer.runtime.bloomFilter.maxNumBits` of it, while `created` holds the
+ // deserialized filter instead. Dropping the expression lets the task reclaim those bytes for
Review Comment:
Qualified. The comment says the reclaim holds for a columnar scan's task, and that a row-based one keeps the bytes through the scan node its closure captures. I added it to the broadcast item in the description as well, since a broadcast handle in the plan instead of the bytes would fix both.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1947,6 +1947,48 @@ object SQLConf {
.checkValue(threshold => threshold >= 0, "The threshold must not be negative.")
.createWithDefault(10)
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED =
+ buildConf("spark.sql.parquet.storageFilterPushdown.enabled")
+ .doc("If true, the vectorized Parquet reader may apply a runtime storage filter, such as a " +
+ "bloom filter from join runtime filtering, while it reads. It reads the columns the " +
+ "filter needs first, evaluates the filter per row, and then reads the remaining columns " +
+ "only for the rows that survived. This is a planning-time decision. With it false, no " +
+ "storage filter is attached to a scan at all. A filter that is attached also stays in " +
+ "the post-scan filter, the way a pushed data filter does, so the reader is free to stop " +
+ "applying it wherever doing so would cost more than it saves, and the answer does not " +
+ "change. Narrowing a row group to part of its rows trusts the Parquet page index, which " +
+ "a read with no pushed data filter never consults, so on a file whose page index is " +
+ "wrong this can pair a row's key with another row's values. Setting " +
+ "parquet.filter.columnindex.enabled to false is the escape hatch, and it leaves the " +
+ "filter with the row groups it empties, which need no index at all. Note that the " +
+ "surviving key values of a whole row group are buffered before that row group's first " +
+ "batch is produced, so a task holds up to one extra copy of the key columns for one row " +
+ "group.")
+ .version("5.0.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
+ val PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES =
+ buildConf("spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes")
+ .internal()
+ .doc("The maximum memory, in bytes, that the vectorized Parquet reader holds for one row " +
+ "group while it applies a storage filter. Two things count against it, and both grow " +
+ "with the number of surviving rows. One is the key values buffered to splice into the " +
+ "output batches, which follow the reader's memory mode and so can be off heap. The other " +
+ "is the row ranges those rows fall into, always on heap, which the second phase needs to " +
+ "select its pages. The count is examined after every surviving row. Past the limit the " +
+ "reader releases the buffer and reads every projected column of the surviving rows " +
+ "instead, which costs one extra read of the key columns. Past it again it reads the row " +
+ "group with no filter applied at all, which is correct but as slow as not pushing the " +
Review Comment:
Fixed in all three places. The budget doc says the give-up pays that same extra key read and so is slower than not pushing the filter. The `enabled` doc says a row group where the reader stops reads its key columns twice. It also says the columnindex escape hatch keeps the row groups the filter empties while every other row group reads its key columns twice, and it names turning the conf off as the other way out. The reader comment says a given-up row group read what a plain scan reads plus the key columns phase 1 had already read. The description's list of give-up paths says the same.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled once the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Set to true once this reader has no rows left to return, which is when the row groups run out
+ * or when the filter rejects the whole file.
+ */
+ private boolean hitEndOfData = false;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch is a view whose slots alias these vectors (non-key and partition) and the head of
+ // each survivor queue (key slots), so closing it would re-close a shared vector and free the
Review Comment:
Rewritten, in the places you name and one more. `closeAllocated`'s comment says the persistent key vectors are what closing the spliced batch would leave unreached. The base class's `closeAllocated` javadoc made the same double-free claim, and it now just says a subclass closes what its batch does not hold. The suite comment says a read after close is the off-heap hazard and that a second close is harmless.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -0,0 +1,1200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.PrimitiveIterator;
+import java.util.Set;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.columnindex.RowRanges;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import org.apache.spark.internal.LogKeys;
+import org.apache.spark.internal.MDC;
+import org.apache.spark.internal.SparkLogger;
+import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.expressions.BasePredicate;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+/**
+ * A {@link VectorizedParquetRecordReader} that reads a row group in phases under a
+ * {@code ParquetStorageFilter}, so the projected columns of the rows the filter rejects are never
+ * read. The filter is a deterministic conjunct of the post-scan {@code Filter}, which still holds
+ * it, so any phase below may stop short and read the file the way a plain scan would.
+ *
+ * <p>Per row group, all three phases driven by the one {@link #fileReader}:
+ * <ul>
+ * <li>phase 0 asks the column index which rows the pushed data filter allows, metadata only;</li>
+ * <li>phase 1 reads the key columns the filter references over those rows, evaluates the filter
+ * per row, and buffers the surviving key values;</li>
+ * <li>phase 2 reads the remaining projected columns over the surviving row ranges.</li>
+ * </ul>
+ *
+ * <p>The emit path then splices the buffered key vectors into the batch, so phase 2 never reads a
+ * key column again. A row group whose buffer would pass
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} gives splicing up, and
+ * past that the filter itself, and is read the plain way.
+ *
+ * <p>Splicing does not engage at all in one real case, when every key column is missing from this
+ * physical file under schema evolution. The predicate is then constant over the file and is
+ * evaluated once, against the constants the reader materializes for those columns. True keeps the
+ * file unfiltered, false skips it.
+ *
+ * <p>Everything else the filter needs is guaranteed by {@code FileSourceStrategy.storageFiltersFor}
+ * and {@code ParquetStorageFilter.create}, so a violation of it here is a planner bug and is
+ * asserted rather than handled. A file the reader simply cannot prune is a different matter, and it
+ * reads that one the way a plain scan would.
+ */
+public class LateMaterializationParquetRecordReader extends VectorizedParquetRecordReader {
+
+ private static final SparkLogger LOG =
+ SparkLoggerFactory.getLogger(LateMaterializationParquetRecordReader.class);
+
+ /**
+ * What one buffered variable-length value costs the vector that holds it, as a multiple of the
+ * value's own bytes. Those bytes go in the byte child, with one null byte per element of that
+ * child, and up to as much again because {@link WritableColumnVector#reserve} doubles the child
+ * when it grows. A factor rather than an exact figure, so this half of the charge over-counts
+ * rather than under-counts, since nothing above this buffer can spill it. A fixed-width key needs
+ * no factor, since its accumulator is allocated at capacity and never grows.
+ */
+ private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
+
+ /**
+ * The storage filter this reader applies. It reads key-column pages first, evaluates the filter
+ * per row to build {@link RowRanges}, then reads data-column pages restricted to those ranges.
+ * Nulled once the reader gives it up for the whole file, after which every path here behaves as
+ * the plain reader does.
+ */
+ private ParquetStorageFilter storageFilter;
+
+ /**
+ * Set to true once this reader has no rows left to return, which is when the row groups run out
+ * or when the filter rejects the whole file.
+ */
+ private boolean hitEndOfData = false;
+
+ /**
+ * Late-materialization state, populated by {@link #initializeLateMaterialization()}.
+ *
+ * <p>The base class's {@link #fileReader} drives all three phases. Its requested
+ * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected
+ * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, and for phase 2 the
+ * non-key columns while splicing or the whole projection once a row group gives it up.
+ *
+ * <p>The three sets are held as leaf-column lists rather than as {@code MessageType}s because
+ * that is what both the reader and the byte metrics consume, and because
+ * {@code MessageType.getColumns()} rebuilds the list on every call.
+ *
+ * <p>Each list's leaf {@link ColumnPath}s are resolved alongside it, because that is all the
+ * footer walks below need and {@link ColumnPath#get} is not free, since it allocates and interns
+ * through parquet's canonicalizer, which those walks would otherwise repeat per column on every
+ * one of their several calls per row group. As sets, because a walk goes over the block's column
+ * chunks and asks which of them it is about, and a requested schema's leaf paths are unique
+ * anyway.
+ */
+ private List<ColumnDescriptor> requestedColumns;
+ private List<ColumnDescriptor> keyOnlyColumns;
+ private List<ColumnDescriptor> nonKeyColumns;
+ private Set<ColumnPath> requestedPaths;
+ private Set<ColumnPath> keyOnlyPaths;
+ private Set<ColumnPath> nonKeyPaths;
+ /**
+ * Whether a row group can be read in part, and what reading part of one transfers. Both are read
+ * off the footer {@link #fileReader} already holds, so the byte metrics cost no IO.
+ */
+ private ParquetBlockAccounting blockAccounting;
+ /**
+ * What an accumulator holds per row for each key column, not counting the value bytes of a
+ * variable-length type. Together with those value bytes it is what the per-row-group buffer is
+ * measured against. See
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}.
+ */
+ private int keyFixedBytesPerRow;
+ /**
+ * The pages phase 2 read for the row group being emitted. Held because the column readers draw
+ * from it for the whole row group, and closed when the next one is loaded, since parquet closes
+ * neither the stores `readFilteredRowGroup` hands back nor the ones from `readNextRowGroup`.
+ */
+ private PageReadStore dataPages;
+
+ /** Whether the filter has already been reported as failing to evaluate on this file. */
+ private boolean loggedFilterEvaluationError;
+
+ /**
+ * Whether {@code parquet.filter.columnindex.enabled} says this file's page index is not to be
+ * used. That covers phase 0 as well as the narrowing below, because the conf is the escape hatch
+ * for a file whose index is wrong, and phase 2 reads part of a row group through the offset index
+ * whatever the conf says. A wrong index there pairs a row's key with another row's values, which
+ * the post-scan Filter cannot catch, since the key it sees is the right one.
+ */
+ private boolean pageIndexDisabled;
+
+ /**
+ * Whether a read of part of a row group has already failed for want of an offset index, which
+ * says this file is not to be read in part again. Only the narrowing goes. The rows the pushed
+ * data filter's own page pruning excludes are still excluded, because parquet built those ranges
+ * itself, and a block it has no index for it hands back whole.
+ */
+ private boolean partialReadUnavailable;
+
+ /**
+ * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to
+ * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass
+ * the cap. False means phase 2 read every projected column, key columns included, so the emit
+ * path takes them straight from the persistent batch.
+ */
+ private boolean spliceCurrentRowGroup;
+ private int nextBlockIndex;
+ /** This file's row groups, as the footer lists them after parquet's own row-group filtering. */
+ private List<BlockMetaData> blocks;
+ /**
+ * The key columns this file has, in the filter's key order, with everything init resolves about
+ * each. Phase 1 reads them, and the survivor queues and accumulators are indexed the same way.
+ */
+ private KeyColumn[] keyColumns;
+ /**
+ * Whether every key column is missing from this file, which makes the predicate constant for it.
+ * Decided in {@link #initBatch}, since the constants only exist once its vectors are built.
+ */
+ private boolean allKeysMissing;
+ private WritableColumnVector[] keyScratchVectors;
+ private ColumnarBatch keyScratchBatch;
+
+ /**
+ * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit
+ * path splices them back into the output batch, so phase 2 never reads the key columns.
+ *
+ * <p>The alternative is for phase 2 to read the key columns again under the surviving row ranges,
+ * which needs none of this state but pays a second read of them for every row group. Nothing
+ * absorbs that read on object storage, where it is a new GET rather than a page-cache hit.
+ *
+ * <p>{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather
+ * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of
+ * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling.
+ * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so
+ * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is
+ * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch
+ * is built over, which is why the two are closed separately.
+ *
+ * <p>Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold
+ * every surviving key value of one row group at once. That is bounded per row group by
+ * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row
+ * group is read the plain way and nothing is buffered.
+ */
+ private boolean[] isKeyTopLevel;
+ private ArrayDeque<WritableColumnVector>[] keyVectorQueues;
+ private WritableColumnVector[] currentKeyAccumulators;
+ /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */
+ private int currentKeyAccumulatorRowCount;
+ /** Whether each queue's head is the vector the current batch is built on. */
+ private boolean keyVectorsPublished;
+ private ColumnVector[] persistentBatchColumns;
+ private ColumnVector[] spliceBatchColumns;
+
+ /**
+ * The filter is a constructor argument rather than a setter, because
+ * {@link #initializeLateMaterialization()} inspects the per-file schema against it and runs from
+ * {@code initialize}: there is no point in the reader's life at which it is useful without one.
+ */
+ public LateMaterializationParquetRecordReader(
+ ZoneId convertTz,
+ String datetimeRebaseMode,
+ String datetimeRebaseTz,
+ String int96RebaseMode,
+ String int96RebaseTz,
+ boolean useOffHeap,
+ int capacity,
+ ParquetStorageFilter storageFilter) {
+ super(convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz,
+ useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ // For test only.
+ public LateMaterializationParquetRecordReader(
+ boolean useOffHeap, int capacity, ParquetStorageFilter storageFilter) {
+ super(useOffHeap, capacity);
+ this.storageFilter = Objects.requireNonNull(storageFilter, "storageFilter");
+ }
+
+ @Override
+ protected void closeAllocated() {
+ // Each release below is independent, and the base class's close() owns the file handle and
+ // input stream, so they are chained through finally blocks, and one failing vector close must
+ // not leak the rest.
+ try {
+ // The batch's vectors are closed through `persistentBatchColumns` rather than through
+ // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted
+ // batch is a view whose slots alias these vectors (non-key and partition) and the head of
+ // each survivor queue (key slots), so closing it would re-close a shared vector and free the
+ // same buffer twice. Without splicing its array is this one.
+ try {
+ if (persistentBatchColumns != null) {
+ closeAll(persistentBatchColumns);
+ } else if (columnarBatch != null) {
+ // `takeOverBatch` has not run, or threw part way, and the batch super built is then the
+ // only thing holding its vectors.
+ columnarBatch.close();
+ }
+ persistentBatchColumns = null;
+ spliceBatchColumns = null;
+ columnarBatch = null;
+ } finally {
+ closeSplicingState();
+ }
+ } finally {
+ try {
+ // Through the array, not through `keyScratchBatch`, which is assigned only after the
+ // allocation loop finishes, so a partial failure leaves vectors only the array can reach.
+ closeAll(keyScratchVectors);
+ keyScratchVectors = null;
+ keyScratchBatch = null;
+ } finally {
+ closeDataPages();
+ }
+ }
+ }
+
+ @Override
+ public float getProgress() {
+ // Under a storage filter, rowsReturned counts survivors while totalRowCount is the count before
+ // the filter narrowed anything, so the ratio neither reaches 1 on its own nor stays below it. A
+ // row group read whole after the page index is given up can return more rows than that count.
+ // `hitEndOfData` is the real terminator here, and the ratio is clamped to what it means.
+ if (hitEndOfData) return 1.0f;
+ return Math.min(1.0f, super.getProgress());
+ }
+
+ /**
+ * Builds the batch the way the plain reader does, takes it over for splicing, and then answers
+ * the filter for a file that has none of its key columns.
+ *
+ * <p>Both after super's body, because the batch it hands out is what the splicing one is built
+ * over, and the all-keys-missing decision evaluates the predicate against the constant vectors
+ * this reader returns for those columns, which exist only once super has wrapped the allocated
+ * vectors in {@link ParquetColumnVector}s.
+ */
+ @Override
+ protected void initBatch(
+ MemoryMode memMode,
+ StructType partitionColumns,
+ InternalRow partitionValues) {
+ super.initBatch(memMode, partitionColumns, partitionValues);
+ takeOverBatch();
+ if (allKeysMissing) {
+ decideAllKeysMissingFile();
+ }
+ }
+
+ /**
+ * Takes the vectors of the batch super built, and while splicing hands out a batch of this
+ * reader's own over a copy of that array, whose key slots the emit path rewrites in place.
+ * {@link ColumnarBatch} holds the array by reference, its staging row included, so rewriting a
+ * slot is what publishes it, and the object being reused is what makes a consumer see it.
+ *
+ * <p>Every slot got a vector, key columns included. While splicing a key slot's vector is
+ * unused, since the emitted batch takes that slot from the survivor queues. A row group read the
+ * plain way past the buffer cap does read into it, and one capacity-sized vector per key column
+ * is cheap next to the buffer the cap is there to bound.
+ */
+ private void takeOverBatch() {
+ persistentBatchColumns = new ColumnVector[columnarBatch.numCols()];
+ for (int i = 0; i < persistentBatchColumns.length; i++) {
+ persistentBatchColumns[i] = columnarBatch.column(i);
+ }
+ if (keyColumns == null) return;
+ spliceBatchColumns = persistentBatchColumns.clone();
+ columnarBatch = new ColumnarBatch(spliceBatchColumns);
+ }
+
+ /**
+ * Answers the storage filter for a file that has none of its key columns, which makes the
+ * predicate constant over the file. It is evaluated against the vectors just built for those
+ * columns, so the value it sees is the one the scan returns for them.
+ *
+ * <p>False skips the file. True, or an error, reads it the way a plain scan would, under the same
+ * fail-open rule as a row's value, since the constant can be one the predicate throws on.
+ */
+ private void decideAllKeysMissingFile() {
+ allKeysMissing = false;
+ // Not closed, and must not be, since the vectors in it belong to the batch this reader emits.
+ ColumnarBatch keyRow = new ColumnarBatch(keyVectorsFromBatch(), 1);
+ boolean keep;
+ try {
+ keep = storageFilter.preparedPredicate().eval(keyRow.getRow(0));
+ } catch (RuntimeException e) {
+ logFilterGivenUpOnError(e);
+ keep = true;
+ }
+ if (!keep) {
+ recordFileSkipped();
+ hitEndOfData = true;
+ }
+ storageFilter = null;
+ }
+
+ /**
+ * Whether this read has returned every row it is going to.
+ *
+ * <p>{@code totalRowCount} is how many rows the file yields before this filter narrows anything,
+ * so it is a terminator only while there is no filter. Under one it is an upper bound the read
+ * never reaches, and a row group read whole after the page index was given up can pass it. What
+ * ends such a read is {@link #hitEndOfData}, set when the row groups run out or when the filter
+ * rejects the whole file.
+ */
+ @Override
+ protected boolean noMoreRows() {
+ return hitEndOfData || (storageFilter == null && super.noMoreRows());
+ }
+
+ /**
+ * Releases the key vectors the previous batch was built on, before the batch is emptied, so a
+ * terminal call cannot leave a spliced batch pointing at vectors just released, which off heap is
+ * freed memory.
+ */
+ @Override
+ protected void resetBatch() {
+ releasePublishedKeyVectors();
+ super.resetBatch();
+ }
+
+ /**
+ * Points the emitted batch's key slots at what this batch takes them from, then reads the rest
+ * the way the plain path does. Three cases reach the base's loop. A spliced row group, whose key
+ * slots have no phase-2 reader because the emitted batch takes them from the survivor queues. A
+ * row group read the plain way past the buffer cap, which has a reader for every slot. And a file
+ * the filter was given up for, which is the plain path exactly.
+ *
+ * <p>The row-index column the base fills there is identified by name
+ * (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, so
+ * its slot is always a non-key one and its ParquetColumnVector is always the persistent one.
+ */
+ @Override
+ protected void readBatchColumns(int num) throws IOException {
+ // Without a storage filter the emitted batch is `initBatch`'s array and this does not apply.
+ if (spliceBatchColumns != null) {
+ pointKeySlotsAtThisBatch(num);
+ }
+ super.readBatchColumns(num);
+ }
+
+ /**
+ * Points the emitted batch's key slots at the vectors this batch takes them from. They are the
+ * only slots that ever change, since a non-key slot holds the vector {@link #initBatch} built for
+ * it for the reader's life. A spliced row group takes the head of each survivor queue, which is
+ * what publishes the buffered keys, and any other row group takes the persistent vector, which
+ * is what a row group following a spliced one needs.
+ *
+ * <p>The same {@link ColumnarBatch} is handed out every time, over an array it holds by
+ * reference. The queues keep owning the vectors they publish until the next batch releases them,
+ * so a phase-2 read that throws afterwards leaves them reachable for {@link #close()}.
+ */
+ private void pointKeySlotsAtThisBatch(int num) {
+ keyVectorsPublished = spliceCurrentRowGroup;
+ // Queue k holds the survivors of key column k, and its `batchSlot` is where that key column
+ // sits in the batch, so the pairing is read off rather than re-derived from slot order.
+ for (int k = 0; k < keyColumns.length; k++) {
+ int slot = keyColumns[k].batchSlot();
+ spliceBatchColumns[slot] =
+ spliceCurrentRowGroup ? nextSurvivorVector(k, num) : persistentBatchColumns[slot];
+ }
+ }
+
+ /** The head of survivor queue {@code k}, which is what publishes the keys it buffered. */
+ private WritableColumnVector nextSurvivorVector(int k, int num) {
+ WritableColumnVector vector = keyVectorQueues[k].peekFirst();
+ if (vector == null) {
+ // Unreachable, since the queues hold exactly the survivors phase 1 accumulated and the emit
+ // loop is driven by that same count. Named here rather than left to a null batch slot and an
+ // NPE somewhere in the read that follows.
+ throw new IllegalStateException(String.format(
+ "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still "
+ + "to emit", k, nextBlockIndex - 1, fileReader.getFile(), num));
+ }
+ return vector;
+ }
+
+ /**
+ * Closes the key vectors the previous batch was built on. The queues own them until here, so
+ * survivor memory drains as a row group is emitted rather than all at its end.
+ */
+ private void releasePublishedKeyVectors() {
+ if (!keyVectorsPublished) return;
+ keyVectorsPublished = false;
+ for (ArrayDeque<WritableColumnVector> queue : keyVectorQueues) {
+ queue.removeFirst().close();
+ }
+ }
+
+ @Override
+ protected void initializeInternal() throws IOException, UnsupportedOperationException {
+ super.initializeInternal();
+ initializeLateMaterialization();
+ }
+
+ private void initializeLateMaterialization() throws IOException {
+ if (fileReader == null) {
+ // The three phases drive a ParquetFileReader directly, and this reader was handed a row-group
+ // reader with none behind it, so the filter is simply not applied and the plain read path
+ // runs. The conjunct is in the post-scan filter as well, so the rows it would have dropped
+ // are dropped above the scan.
+ storageFilter = null;
+ return;
+ }
+ blockAccounting = new ParquetBlockAccounting(fileReader);
+ blocks = fileReader.getRowGroups();
+ // Both of the flags the narrowing hangs on start here. The conf one closes phase 0 too, and the
+ // other is what the failed read below latches. Neither costs the filter the row groups it
+ // empties, which need no index at all.
+ pageIndexDisabled =
+ !configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true);
+ // Resolve each key column's top-level ParquetColumn, keeping the ones this file has. A missing
+ // one can happen under schema evolution, where a column is in the requested schema but not in
+ // the file. For any non-primitive key we still bail, since phase 1 reads only primitive leaves.
+ int[] keyIndices = storageFilter.keyColumnIndices();
+ List<ParquetColumn> presentKeyColumns = new ArrayList<>(keyIndices.length);
+ List<Integer> presentKeyPositions = new ArrayList<>(keyIndices.length);
+ for (int i = 0; i < keyIndices.length; i++) {
+ int idx = keyIndices[i];
+ if (idx < 0 || idx >= parquetColumn.children().size()) {
+ // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals.
+ throw new IllegalStateException(String.format(
+ "Storage-filter key ordinal %d is out of range for a %d-column requested schema",
+ idx, parquetColumn.children().size()));
+ }
+ ParquetColumn column = parquetColumn.children().apply(idx);
+ if (!column.isPrimitive()) {
+ // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive
+ // Parquet leaf, and it gates both planning and ParquetStorageFilter.create.
+ throw new IllegalStateException(
+ "Storage-filter key column is not a primitive Parquet column: " + column.path());
+ }
+ if (!missingColumns.contains(column)) {
+ presentKeyPositions.add(i);
+ presentKeyColumns.add(column);
+ }
+ }
+
+ // A key column this file does not have is not substituted into the predicate. Its slot in the
+ // row phase 1 evaluates points at the vector `initBatch` builds for it, which holds exactly
+ // what the scan will return for that column, its existence DEFAULT or null where it has none.
+ // So there is one rule for what a missing column reads as, ParquetColumnVector's, and the
+ // filter cannot decide on a value the rows never show.
+ requestedColumns = requestedSchema.getColumns();
+ requestedPaths = ParquetBlockAccounting.pathsOf(requestedColumns);
+ if (presentKeyColumns.isEmpty()) {
+ // Nothing for phase 1 to read, so the predicate is constant for this file. Deciding needs the
+ // vectors `initBatch` has not built yet, so it happens there, in `decideAllKeysMissingFile`.
+ allKeysMissing = true;
+ return;
+ }
+
+ int numKeys = presentKeyColumns.size();
+ ColumnDescriptor[] keyDescriptors = new ColumnDescriptor[numKeys];
+ Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage();
+ Set<String> keyTopLevelNames = new HashSet<>();
+ for (int i = 0; i < numKeys; i++) {
+ keyDescriptors[i] = presentKeyColumns.get(i).descriptor().get();
+ // Preserve the field name/type as it appears at the top of requestedSchema.
+ String topLevelName = keyDescriptors[i].getPath()[0];
+ keySchemaBuilder.addField(requestedSchema.getType(topLevelName));
+ keyTopLevelNames.add(topLevelName);
+ }
+ keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns();
+ keyOnlyPaths = ParquetBlockAccounting.pathsOf(keyOnlyColumns);
+
+ // Build the non-key (complement) schema, which phase 2 reads under finalRanges. With nothing in
+ // it there is nothing to prune, since such a scan reads the same columns for the same rows as a
+ // plain one, phase 1 having to read a key column to evaluate the filter on it. So the reader
+ // declines rather than carrying a second emit path for it. The planner does not push this shape
+ // either, which makes the two consistent without depending on each other.
+ Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage();
+ int nonKeyFieldCount = 0;
+ for (Type field : requestedSchema.getFields()) {
+ if (!keyTopLevelNames.contains(field.getName())) {
+ nonKeyBuilder.addField(field);
+ nonKeyFieldCount++;
+ }
+ }
+ if (nonKeyFieldCount == 0) {
+ storageFilter = null;
+ return;
+ }
+ nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns();
+ nonKeyPaths = ParquetBlockAccounting.pathsOf(nonKeyColumns);
+ // Resolved here, past the two returns above, because `ValueCopier.forType` rejects a type the
+ // reader cannot copy, and a file the reader has already declined must not be failed over one.
+ keyColumns = new KeyColumn[numKeys];
+ keyFixedBytesPerRow = 0;
+ StructField[] fields = sparkRequestedSchema.fields();
+ for (int i = 0; i < numKeys; i++) {
+ int rowPosition = presentKeyPositions.get(i);
+ int batchSlot = keyIndices[rowPosition];
+ DataType type = fields[batchSlot].dataType();
+ boolean variableLength = ValueCopier.isVariableLength(type);
+ keyColumns[i] = new KeyColumn(batchSlot, rowPosition, keyDescriptors[i],
+ presentKeyColumns.get(i).required(), type, ValueCopier.forType(type), variableLength);
+ // One null byte per row either way. A fixed-width value adds its own width. A variable-length
+ // one adds the int offset and int length that point at the byte child, plus what the child
+ // itself is allocated per row before a value is written, which is four bytes of data, one per
+ // WritableColumnVector.DEFAULT_ARRAY_LENGTH, and a null byte for each of them.
+ keyFixedBytesPerRow += variableLength ? 1 + 4 + 4 + 4 + 4 : 1 + type.defaultSize();
+ }
+ initializeSplicingState();
+ }
+
+ /**
+ * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level
+ * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map
+ * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from
+ * the survivor queues.
+ */
+ @SuppressWarnings("unchecked")
+ private void initializeSplicingState() {
+ isKeyTopLevel = new boolean[sparkSchema.fields().length];
+ // Only the key columns this file has, since a missing one keeps the constant vector `initBatch`
+ // built for its slot, there being nothing to buffer and nothing to splice back.
+ for (KeyColumn key : keyColumns) {
+ isKeyTopLevel[key.batchSlot()] = true;
+ }
+ // Both arrays before either is filled, so `closeSplicingState` finding one of them finds both,
+ // whatever failed part way through here.
+ keyVectorQueues = new ArrayDeque[keyColumns.length];
+ currentKeyAccumulators = new WritableColumnVector[keyColumns.length];
+ for (int i = 0; i < keyColumns.length; i++) {
+ keyVectorQueues[i] = new ArrayDeque<>();
+ }
+ currentKeyAccumulatorRowCount = 0;
+ }
+
+ /**
+ * Loads the next row group using the three-phase late-materialization pattern, all driven by the
+ * one {@link #fileReader} with its requested schema mutated per phase:
+ * - Phase 0, over the full schema, computes {@code pushedFilterRanges} from the pushed data
+ * filter via the column index, metadata only, using
+ * {@link ParquetFileReader#getRowRanges}.
+ * - Phase 1, over the key-only schema, reads key-column pages restricted to
+ * {@code pushedFilterRanges}, evaluates the storage filter per row and builds
+ * {@code finalRanges}.
+ * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave
+ * splicing up reads the whole projection instead, still under {@code finalRanges}, and one
+ * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain
+ * scan reads.
+ *
+ * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). False
+ * says the row groups have run out, which is what ends a read the filter narrowed.
+ */
+ @Override
+ protected boolean loadNextRowGroup() throws IOException {
+ // Nulled once the reader gave the filter up for the whole file, and then the plain path is
+ // exactly what this read wants.
+ if (storageFilter == null) return super.loadNextRowGroup();
+ // The previous row group is fully emitted by the time this is called, so its pages are done
+ // with. Released here rather than at the next assignment, so an all-keys row group, which reads
+ // no data pages at all, does not keep the one before it alive.
+ closeDataPages();
+ while (nextBlockIndex < blocks.size()) {
+ int blockIdx = nextBlockIndex++;
+ long blockRowCount = blocks.get(blockIdx).getRowCount();
+ if (blockRowCount == 0) {
+ // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build
+ // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too.
+ continue;
+ }
+ // Splicing buffers one key value per surviving row of the whole row group before it can emit
+ // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it
+ // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold.
+ // Past that it gives splicing up, and past it again the filter itself.
+
+ // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full
+ // requestedSchema goes back on first, because phases 1 and 2 narrow it and
+ // ParquetFileReader.getRowRanges computes ranges against the reader's current paths.
+ fileReader.setRequestedSchema(requestedColumns);
+ RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount);
+ // RowRanges.rowCount() walks every range, so resolve each range set's count once.
+ long baselineRows = pushedFilterRanges.rowCount();
+ if (baselineRows == 0) {
+ // Pushed data filter rejects this block entirely via column index. Not a storage-filter
+ // skip, so we don't increment storage-filter metrics.
+ continue;
+ }
+
+ StorageFilterMetrics m = storageFilter.metrics();
+
+ // Whether phase 2 can read part of this row group at all, which is what everything below
+ // buffers for. Reading part of one goes through the block's column index store, and parquet
+ // empties that store altogether if one column it was built over has no offset index. Which
+ // columns those are depends on when it is first built. With a pushed data filter it is phase
+ // 0's `getRowRanges` that builds it, over the whole projection, and a key column without an
+ // offset index is then enough to make phase 2 throw. So the question is asked of the whole
+ // projection, which also makes the answer the same for every row group of a file whatever
+ // else is pushed. It is stricter than the read needs with no data filter pushed, where the
+ // store would be built over the non-key columns alone, and such a row group loses its page
+ // filtering for nothing. The shape is a file whose columns do not all carry an offset index,
+ // which no writer produces by itself.
+ // False says the filter is left with emptying this row group whole, and then nothing is
+ // buffered either, and phase 2 reads the key columns again along with everything else, over
+ // ranges an empty store has already widened to the whole block, which is why it cannot throw.
+ // Parquet's own exception stays as the backstop for the one shape the footer cannot see, an
+ // offset index it claims but cannot produce.
+ boolean canNarrowRowGroup = !pageIndexDisabled && !partialReadUnavailable
+ && blockAccounting.hasOffsetIndexes(blockIdx, requestedPaths);
+ spliceCurrentRowGroup = canNarrowRowGroup;
+
+ // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the
+ // storage filter per row. The defaults below are what a row group whose filter is given up
+ // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would.
+ RowRanges finalRanges = pushedFilterRanges;
+ long finalRowCount = baselineRows;
+ fileReader.setRequestedSchema(keyOnlyColumns);
+ // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store
+ // the file reader does not track, unlike `readNextRowGroup`, so nothing else would.
+ RowRanges survivors;
+ try (PageReadStore keyPages =
+ fileReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) {
+ if (keyPages == null) {
+ // Unreachable, since readFilteredRowGroup returns null only for an empty block or empty
+ // ranges, and this block has rows the pushed filter kept. Should parquet ever say
+ // otherwise, the filter is given up for this row group, which reads what a plain read
+ // reads. The alternative is to fail a query a plain read answers.
+ abandonSplicing();
+ survivors = null;
+ } else {
+ survivors = evaluateStorageFilter(
+ keyPages, pushedFilterRanges, baselineRows, blockRowCount, canNarrowRowGroup);
+ }
+ }
+ // Null says phase 1 gave the filter up for this row group, and then the ranges it built are
+ // incomplete, and what gets read is every row the pushed filter kept, exactly what a plain
+ // read would read. Phase 2 can give it up too, in the catch below.
+ boolean filterGivenUp = survivors == null;
+ // What this feature can avoid reading is the non-key columns of the rows the storage filter
+ // rejects, so that is the baseline the byte metrics are measured against, the non-key bytes a
+ // plain read of this projection would transfer for every row the pushed filter kept. Nothing
+ // is measured for a row group whose filter was given up, so nothing is walked for it either.
+ // compressedBytesForRowRanges never does IO of its own.
+ long nonKeyBaselineBytes = filterGivenUp ? 0L : blockAccounting.compressedBytesForRowRanges(
+ blockIdx, nonKeyPaths, pushedFilterRanges, baselineRows);
+ if (survivors != null) {
+ finalRanges = survivors;
+ finalRowCount = survivors.rowCount();
+ if (finalRowCount == 0) {
+ // Every surviving row was rejected by the storage filter, so the block is skipped
+ // entirely, which avoids the whole non-key baseline. Phase 1 still paid to read the key
+ // columns, and that cost is not part of the baseline, so nothing is subtracted from it
+ // here.
+ m.recordRowGroupSkipped(baselineRows, nonKeyBaselineBytes);
+ continue;
+ }
+ }
+
+ // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under
+ // `pushedFilterRanges` for a row group whose filter was given up.
+ fileReader.setRequestedSchema(spliceCurrentRowGroup ? nonKeyColumns : requestedColumns);
+ // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet
+ // enforces that itself, resolving every requested column's offset index before it reads
+ // anything, and a column without one makes its column index store throw
+ // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that
+ // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have
+ // none. The filter is then given up for this row group and the read retried over
+ // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same
+ // wall, because a store missing one column's offset index reports no column index either, so
+ // `getRowRanges` could not have narrowed anything and the ranges cover the whole block.
+ //
+ // The footer check above answers every shape a writer produces, so what is left here is an
+ // index the footer claims and parquet cannot then produce. This is also where such a throw
+ // costs least, landing before any data page is read.
+ try {
+ dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
+ } catch (MissingOffsetIndexException e) {
+ // The exception's own message rather than the exception, since this is a file shape the
+ // feature expects and degrades on, so a stack trace per file of a table written without a
+ // page index is noise. Parquet logs the same condition on its own path without one either.
+ LOG.warn("Reading {} without page-level storage filtering. Reading part of a row group "
+ + "needs a Parquet offset index, and this file has none for at least one column the "
+ + "read needs ({}). Row groups the filter empties are still skipped whole",
+ MDC.of(LogKeys.PATH, fileReader.getFile()),
+ MDC.of(LogKeys.REASON, e.getMessage()));
+ // Only the narrowing is given up for the rest of the file. The pushed data filter's own
+ // page pruning stays, because parquet built those ranges itself and a block it has no index
+ // for it hands back whole, and dropping it here would read more than a plain scan does.
+ partialReadUnavailable = true;
+ filterGivenUp = true;
+ abandonSplicing();
+ finalRanges = pushedFilterRanges;
+ finalRowCount = baselineRows;
+ if (baselineRows != blockRowCount) {
Review Comment:
Done as you suggest. The widening branch is gone, and so are the `noMoreRows()` and `getProgress()` overrides and the `hitEndOfData` flag. `rowsReturned <= totalRowCount` holds on every path now, so the base terminator stays and `nextBatch` is back to master's check. A narrowed read ends when `loadNextRowGroup` runs out of row groups, and a file the filter rejects whole sets `totalRowCount` to zero.
The retry now depends on the parquet behavior you traced, that a store missing an offset index leaves the ranges whole. If a later parquet broke that, the retry would throw rather than read more rows.
`storageFilter`'s javadoc now says it is nulled only before the first row group is loaded, and that nulling it later would restart the read at the first row group.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ValueCopier.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.*;
+
+/**
+ * Copies one key value between column vectors. Picked per key column at init time by
+ * {@link #forType(DataType)} and called per surviving row; the caller handles null sources.
+ *
+ * <p>Nothing here is about Parquet. A copier is chosen from the Spark type alone, and both sides of
+ * the copy are {@link WritableColumnVector}s.
+ *
+ * <p>The types this handles are also the types a storage filter's key columns may have:
+ * {@code ParquetStorageFilter.isSupportedKeyType} answers from {@link #supports}, so the planner's
+ * gate, the filter's own checks and the copy itself read one list. That list is narrower than
Review Comment:
Named in the javadoc as a third exclusion. It gives the two-child layout as the reason and says what you describe, that such a filter is not offered and the scan reads as it would without the feature. I left the copiers for later, since the types sit behind `spark.sql.timestampNanosTypes.enabled` and each needs its own byte charge and tests.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -151,6 +151,66 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
}
}
+ /**
+ * The conjuncts of `afterScanFilters` the file format can evaluate at the storage layer for late
+ * materialization, to prune the IO of the columns the filter does not reference.
+ *
+ * They stay in the post-scan `Filter` as well, the way a pushed data filter does. The reader is
+ * offered them, not obliged to honor them, so the plan keeps the exact check. What it costs is
+ * evaluating the conjunct a second time for the rows the scan emits. What it buys is a reader
+ * free to give up on a file it cannot read in part, or on a stretch of rows whose survivors cost
+ * too much to keep track of, without the answer depending on it.
+ *
+ * Two of the conditions are per scan, and failing either offers nothing:
+ * - [[FileFormat.supportsStorageFilterPushdown]] holds. That is where a format reads the conf
+ * that enables this, so a format's own conf never decides for another format, and asking it
+ * first keeps everything below off the path of a scan that will not use it.
+ * - [[FileFormat.supportBatch]] holds for the schema the reader will see,
+ * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives
+ * its own vectorized-read decision from. Late materialization needs a batch read, so this asks
+ * about batch support rather than naming a format.
+ *
+ * The rest are per conjunct:
+ * - It is deterministic. The reader drops the rows the conjunct rejects and the post-scan
+ * `Filter` evaluates it again on the rows it keeps, so the two evaluations have to agree. Such
+ * a conjunct would also fail at task time, since a reader evaluates the predicate without
Review Comment:
Removed that sentence. The deterministic check keeps the reason in its first sentence, and the suite comment gives that same reason now instead of naming `ParquetStorageFilter.test`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-6011559138
@sunchao thank you for the review, and for the note on Comet.
- The cancelled CI shards could not get a hosted runner. The new push runs them again, `ParquetStorageFilterSuite` included.
- P3: done. `storageFiltersFor` takes the filters as they are and builds the `ExpressionSet` only after its two guards, still before the `filter`. A new planner test pins the deduplication, which nothing covered before.
- The benchmark now checks, untimed, that both arms return the same rows, by count and by a fingerprint over every column.
- The pre-sizing is gone, as dongjoon also suggested in this round.
- The coverage boundaries in your table make sense as follow-ups. I'd keep them out of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58895:
URL: https://github.com/apache/spark/pull/58895#issuecomment-6036959477
Thank you @dongjoon-hyun, @sunchao, @cloud-fan and @pingz-oai for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org