You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Aayush7352 (via GitHub)" <gi...@apache.org> on 2026/07/06 04:43:58 UTC
[PR] [SPARK-57739][SQL] Return NULL for Parquet MIN/MAX aggregate push-down over all-null row groups [spark]
Aayush7352 opened a new pull request, #57030:
URL: https://github.com/apache/spark/pull/57030
### What changes were proposed in this pull request?
This PR fixes Parquet MIN/MAX aggregate push-down so that an all-null row group contributes `NULL` (i.e. nothing) to the aggregate instead of failing the query, and an entirely-null column yields `NULL` as MIN/MAX results:
- `ParquetUtils.getCurrentBlockMaxOrMin` now distinguishes two cases when column statistics have no min/max (`!statistics.hasNonNullValue`):
- **All-null block** (`statistics.isNumNullsSet && statistics.getNumNulls == valueCount`): returns `null`, meaning the block contributes nothing to MIN/MAX - mirroring `OrcUtils.getMinMaxFromColumnStatistics`, which returns `null` when `getNumberOfValues == 0`.
- **Genuinely missing statistics**: still throws `PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_MIN_MAX` (e.g. files written with `parquet.column.statistics.enabled=false`), preserving the behavior guarded by the SPARK-57746 test.
- The MIN/MAX accumulation loop in `getPushedDownAggResult` skips `null` block contributions, and appends `null` (instead of the `None` sentinel) when no block contributed a value.
- `createAggInternalRowFromFooter` leaves the corresponding result field `NULL` when the pushed-down value is `null` (fields of `SpecificInternalRow` start as `NULL`).
### Why are the changes needed?
`MIN`/`MAX` over a column whose values are all `NULL` must return `NULL` per SQL semantics. Before this change, Parquet aggregate push-down threw an error as soon as **any** row group of the column was all-null:
```scala
spark.range(10).selectExpr("CAST(NULL AS LONG) AS v").write.parquet("/tmp/t")
spark.conf.set("spark.sql.parquet.aggregatePushdown", true)
spark.read.parquet("/tmp/t").selectExpr("MIN(v)").show()
```
fails with:
```
[PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_MIN_MAX] ... No min/max value found in the column statistics. SQLSTATE: 0A000
```
while the same query succeeds and returns `NULL` with push-down disabled, and with the ORC data source (which already treats an all-null stripe as "no contribution"). Fixes SPARK-57739.
### Does this PR introduce _any_ user-facing change?
Yes. With `spark.sql.parquet.aggregatePushdown=true`, MIN/MAX aggregation over a Parquet column containing an all-null row group previously failed with `PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_MIN_MAX`; it now returns the correct result (`NULL` when the whole column is null), consistent with push-down disabled and with ORC. Queries on files with genuinely missing statistics still fail with the same error as before.
### How was this patch tested?
Two new tests in `FileSourceAggregatePushDownSuite`, which run across Parquet V1/V2 and ORC V1/V2:
- MIN/MAX of an all-null column return `NULL` (alongside a non-null column, COUNT unchanged).
- An all-null file and a non-null file in the same table combine correctly (all-null block contributes nothing).
Both tests fail on master with `PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_MIN_MAX` in the Parquet suites and pass with this change:
```
build/sbt 'sql/testOnly *AggregatePushDownSuite'
...
Tests: succeeded 105, failed 0, canceled 0, ignored 0, pending 0
```
This includes the existing SPARK-57746 test verifying that files written with statistics disabled still raise `NO_MIN_MAX`/`NO_NUM_NULLS`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.7)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57739][SQL] Return NULL for Parquet MIN/MAX aggregate push-down over all-null row groups [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #57030: [SPARK-57739][SQL] Return NULL for Parquet MIN/MAX aggregate push-down over all-null row groups
URL: https://github.com/apache/spark/pull/57030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57739][SQL] Return NULL for Parquet MIN/MAX aggregate push-down over all-null row groups [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #57030:
URL: https://github.com/apache/spark/pull/57030#issuecomment-6181653490
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org