[jira] [Updated] (SPARK-59551) Avoid recomputing existence default values per row in the Parquet, ORC, and CSV readers

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-59551:
-----------------------------------
    Labels: pull-request-available  (was: )

> Avoid recomputing existence default values per row in the Parquet, ORC, and CSV readers
> ---------------------------------------------------------------------------------------
>
>                 Key: SPARK-59551
>                 URL: https://issues.apache.org/jira/browse/SPARK-59551
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.1.0
>            Reporter: David Mollitor
>            Priority: Minor
>              Labels: pull-request-available
>
> h2. Summary
> When a row-based reader materializes each record it applies "existence default values" (the value used for a column that is missing from the data but has a DEFAULT in the schema). The readers do this by calling {{ResolveDefaultColumns.applyExistenceDefaultValuesToRow(schema, row, bitmask }}and {{{}resetExistenceDefaultsBitmask(schema, bitmask){}}}) once per record, and those methods call {{existenceDefaultValues(schema)}} internally:
> {code:scala}
> def existenceDefaultValues(schema: StructType): Array[Any] =
>   schema.fields.map(getExistenceDefaultValue)   // allocates a new Array[Any]; for any field with
>                                                  // a default, parses + evaluates its expression
> {code}
> This is a pure function of the schema (it depends only on field metadata, never on row data) and is not cached, so it is recomputed on every record. {{applyExistenceDefaultValuesToRow}} recomputes it
> and also calls {{{}hasExistenceDefaultValues(schema){}}}, which recomputes it again. The utility even documents this hazard: "The caller should avoid using such methods in a loop for efficiency." Because the recompute happens whether or not the schema actually has any defaults, even the common no-default read pays the per-record array allocation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org