[jira] [Created] (SPARK-58735) "size(array)" forces full read of nested array columns and prevents column pruning for Parquet/Delta arrays

["Eyck Troschke (Jira)" <ji...@apache.org>]

Eyck Troschke created SPARK-58735:
-------------------------------------

             Summary: "size(array)" forces full read of nested array columns and prevents column pruning for Parquet/Delta arrays
                 Key: SPARK-58735
                 URL: https://issues.apache.org/jira/browse/SPARK-58735
             Project: Spark
          Issue Type: Improvement
          Components: Optimizer
    Affects Versions: 4.1.2
         Environment: Apache Spark 4.1.2
Databricks Runtime (Photon)
            Reporter: Eyck Troschke


When applying `size()` to a nested array column stored in Parquet/Delta, Spark reads the *entire* array column including all child struct fields, even though only array offsets / repetition levels are required to compute the array length. This results in significantly higher I/O compared to semantically equivalent queries using `explode()` or accessing smaller sub‑arrays.

This behavior occurs in both Apache Spark and Databricks runtimes. It appears to be caused by conservative Catalyst planning for nested arrays, which prevents column pruning and page skipping.

*Example:*
Table size: ~100GB, ~18B rows  
Column: `events` (ARRAY<STRUCT<...>>)  
Sub‑column: `events.flag_a` (small ARRAY)

```
select sum(size(events));               -- 49s, ~100GB read
select sum(size(events.flag_a));     -- 8s, ~6GB read
select count(*) from t lateral view explode(events);           -- 9s, ~10GB read
select count(*) from t lateral view explode(events.flag_a); -- 8s, ~6GB read
```

Although these queries are semantically equivalent, `size(array)` triggers a full scan of all nested array fields, while `explode(array)` allows column pruning.

*Expected Behavior:*
`size(array)` should only require reading array offsets / repetition levels and should not force full read of all nested struct fields.

*Actual Behavior:*
Spark reads all nested array child columns, causing large and unnecessary I/O.

*Impact:*
Severe performance degradation for large nested arrays in Parquet/Delta tables.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org