You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sunchao (via GitHub)" <gi...@apache.org> on 2026/06/05 17:19:20 UTC

[PR] [SPARK-57176][SQL][4.x] Extend nested column pruning through array-returning functions [spark]

sunchao opened a new pull request, #56345:
URL: https://github.com/apache/spark/pull/56345

   ### Why are the changes needed?
   
   This is the `branch-4.x` backport of #56227, merged to `master` as
   `042ad7d0c4ac1c4d3e9fdeb48e2695fdeb861135`.
   
   [SPARK-57176](https://issues.apache.org/jira/browse/SPARK-57176) follows
   [SPARK-57022](https://issues.apache.org/jira/browse/SPARK-57022), which added nested column
   pruning for `transform` over `array<struct>` inputs.
   
   Array-returning functions still retain the complete input element struct even when downstream
   expressions and lambdas only require a subset of nested fields. For example:
   
   ```sql
   SELECT filter(friends, friend -> friend.last = 'Smith').first
   FROM contacts
   ```
   
   If `friends` contains `first`, `middle`, and `last`, Spark reads all three fields even though the
   query only requires `first` and `last`.
   
   ### What changes were proposed in this PR?
   
   - Merge downstream result-field requirements with lambda requirements for `filter` and
     comparator-based `array_sort`.
   - Propagate projected element schemas through `reverse`, `shuffle`, `slice`, and `array_compact`.
   - Rewrite bound lambda variable types and nested field ordinals after pruning.
   - Retain the complete element schema when the whole result is used, when a lambda consumes the
     whole element, or when default `array_sort` natural ordering requires the full struct.
   
   The conflict resolution is adapted to the transform-only SPARK-57022 implementation currently on
   `branch-4.x`. It does not include the separate SPARK-57175 optimization for `exists` and `forall`.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. Eligible queries using array-returning functions over arrays of structs can read a narrower
   input schema. Query results and SQL APIs are unchanged.
   
   ### How was this patch tested?
   
   - `JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home PATH=/opt/homebrew/opt/openjdk@17/bin:$PATH build/sbt "catalyst/testOnly org.apache.spark.sql.catalyst.expressions.SchemaPruningSuite" "sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetV1SchemaPruningSuite org.apache.spark.sql.execution.datasources.parquet.ParquetV2SchemaPruningSuite org.apache.spark.sql.execution.datasources.orc.OrcV1SchemaPruningSuite org.apache.spark.sql.execution.datasources.orc.OrcV2SchemaPruningSuite -- -z Array" catalyst/scalastyle sql/scalastyle`
     - Catalyst: 11 tests passed.
     - Parquet/ORC datasource suites: 260 tests passed.
     - Catalyst and SQL scalastyle: no errors or warnings.
   - `git diff --check`
   - ASCII scan of all changed Scala files
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Codex (GPT-5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57176][SQL][4.X] Extend nested column pruning through array-returning functions [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #56345:
URL: https://github.com/apache/spark/pull/56345#issuecomment-4656309276

   Merged to `branch-4.x`. Thanks for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57176][SQL][4.X] Extend nested column pruning through array-returning functions [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed pull request #56345: [SPARK-57176][SQL][4.X] Extend nested column pruning through array-returning functions
URL: https://github.com/apache/spark/pull/56345


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org