[jira] [Updated] (SPARK-32731) Add tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse

["Muhammad Samir Khan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-32731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Muhammad Samir Khan updated SPARK-32731:
----------------------------------------
    Summary: Add tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse  (was: Added tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse)

> Add tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse
> ------------------------------------------------------------------------------------
>
>                 Key: SPARK-32731
>                 URL: https://issues.apache.org/jira/browse/SPARK-32731
>             Project: Spark
>          Issue Type: Test
>          Components: SQL, Tests
>    Affects Versions: 3.0.0
>            Reporter: Muhammad Samir Khan
>            Priority: Major
>
> Splitting tests originally posted in [PR|[https://github.com/apache/spark/pull/29352]] for SPARK-32531. The added tests cover cases for maps and arrays of nested structs for different file formats. Eg, [https://github.com/apache/spark/pull/29353] and [https://github.com/apache/spark/pull/29354] add object reuse when reading ORC and Avro files. However, for dynamic data structures like arrays and maps, we do not know just by looking at the schema what the size of the data structure will be so it has to be allocated when reading the data points. The added tests provide coverage so that objects are not accidentally reused when encountering maps and arrays.
> AFAIK this is not covered by existing tests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org