[jira] [Created] (SPARK-60066) Nested UpdateFields expressions duplicate source expressions

["Ben Hollis (Jira)" <ji...@apache.org>]

Ben Hollis created SPARK-60066:
----------------------------------

             Summary: Nested UpdateFields expressions duplicate source expressions
                 Key: SPARK-60066
                 URL: https://issues.apache.org/jira/browse/SPARK-60066
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.4.0
            Reporter: Ben Hollis


Spark represents `Column.withField` as the Catalyst expression `UpdateFields`. Before execution, Spark replaces that expression with a new struct containing the updated value and a read for every unchanged field. Each unchanged-field read contains the complete expression that produces the input struct.
 
When several updates are nested, the input to one replacement contains the replacement generated for the previous update. Spark repeats that entire earlier expression in every unchanged-field read. The logical expression can therefore grow much faster than the number of requested updates, increasing optimizer, expression-binding, and code-generation costs.
 
### Example
{code:java}
val updated = col("s")
.withField("nested.a", lit(1))
.withField("nested.b", lit(2))
.withField("nested.c", lit(3))


df.select(updated) {code}
 
Suppose `s` has the schema `struct<nested:struct<value:int>,d:int,e:int,f:int>`. The query changes three fields under `nested`; it does not change `d`, `e`, or `f`. Spark nevertheless generates reads for those unchanged fields at each replacement layer. Those reads repeat the expression generated for the earlier layer, so the Catalyst tree grows by far more than the three requested changes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org