You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "hemanthboyina (via GitHub)" <gi...@apache.org> on 2026/09/21 13:29:04 UTC
[PR] [SPARK-59609][SQL] Fix Parquet/ORC aggregate push-down reading statistics from the wrong column [spark]
hemanthboyina opened a new pull request, #58945:
URL: https://github.com/apache/spark/pull/58945
**What changes were proposed in this pull request?**
When aggregate push-down is enabled, Spark answers MIN/MAX/COUNT from a file's footer statistics instead of reading the data. To locate a column's statistics it used the column's position in the read schema, then applied that position to the file's physical columns. This is only correct when every file's layout matches the read schema. This PR resolves the column against each file's own schema instead:
for Parquet, it looks the column up by name in the file's footer schema;
for ORC, it resolves the column to its position in the file's own schema, following the same name/positional rules ORC already uses in requestedColumnIds.
If a file does not contain the column, `MIN`/`MAX` return `null` and `COUNT(col)` counts no non-null values, instead of reading the wrong column.
**Why are the changes needed?**
The old position-based lookup silently returns wrong results when a file's column layout differs from the read schema, e.g. Parquet with mergeSchema=true where some files are missing the column, or ORC files with a different column order.
**Does this PR introduce any user-facing change?**
Yes. With aggregate push-down enabled (spark.sql.parquet.aggregatePushdown / spark.sql.orc.aggregatePushdown, both false by default), the queries above now return correct results instead of silently wrong ones.
**How was this patch tested?**
Added tests to FileSourceAggregatePushDownSuite (Parquet mergeSchema with a missing column; ORC with a different column order), run with the vectorized reader on and off. Both fail before the fix and pass after. All existing push-down suites still pass.
**Was this patch authored or co-authored using generative AI tooling?**
Yes, used Claude code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59609][SQL] Fix Parquet/ORC aggregate push-down reading statistics from the wrong column [spark]
Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #58945:
URL: https://github.com/apache/spark/pull/58945#discussion_r4068884033
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala:
##########
@@ -323,54 +326,62 @@ object ParquetUtils extends Logging {
private[sql] def getPushedDownAggResult(
footer: ParquetMetadata,
filePath: String,
- dataSchema: StructType,
partitionSchema: StructType,
- aggregation: Aggregation)
+ aggregation: Aggregation,
+ aggSchema: StructType)
: (Array[PrimitiveType], Array[Any]) = {
val footerFileMetaData = footer.getFileMetaData
- val fields = footerFileMetaData.getSchema.getFields
+ // Resolve by name in the file's own schema; a positional lookup breaks under mergeSchema.
+ val fileSchema = footerFileMetaData.getSchema
+ def fileFieldIndex(colName: String): Int =
+ if (fileSchema.containsField(colName)) fileSchema.getFieldIndex(colName) else -1
Review Comment:
done, fixed it now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59609][SQL] Fix Parquet/ORC aggregate push-down reading statistics from the wrong column [spark]
Posted by "tdcmeehan (via GitHub)" <gi...@apache.org>.
tdcmeehan commented on code in PR #58945:
URL: https://github.com/apache/spark/pull/58945#discussion_r4067826480
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala:
##########
@@ -323,54 +326,62 @@ object ParquetUtils extends Logging {
private[sql] def getPushedDownAggResult(
footer: ParquetMetadata,
filePath: String,
- dataSchema: StructType,
partitionSchema: StructType,
- aggregation: Aggregation)
+ aggregation: Aggregation,
+ aggSchema: StructType)
: (Array[PrimitiveType], Array[Any]) = {
val footerFileMetaData = footer.getFileMetaData
- val fields = footerFileMetaData.getSchema.getFields
+ // Resolve by name in the file's own schema; a positional lookup breaks under mergeSchema.
+ val fileSchema = footerFileMetaData.getSchema
+ def fileFieldIndex(colName: String): Int =
+ if (fileSchema.containsField(colName)) fileSchema.getFieldIndex(colName) else -1
Review Comment:
It seems we don't check case sensitivity here. If so, I believe we need to pass down that flag, [same as the ORC reader](https://github.com/apache/spark/pull/58945/changes#diff-e14fd8725cf71eee7b34fa299c2f3abe5a0033f9abce9de4c7e081ba57991b0bR589).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org