You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Muhammad Samir Khan (Jira)" <ji...@apache.org> on 2020/08/28 21:51:00 UTC
[jira] [Created] (SPARK-32731) Added tests for arrays/maps of
nested structs to ReadSchemaSuite to test structs reuse
Muhammad Samir Khan created SPARK-32731:
-------------------------------------------
Summary: Added tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse
Key: SPARK-32731
URL: https://issues.apache.org/jira/browse/SPARK-32731
Project: Spark
Issue Type: Test
Components: SQL, Tests
Affects Versions: 3.0.0
Reporter: Muhammad Samir Khan
Splitting tests originally posted in [PR|[https://github.com/apache/spark/pull/29352]] for SPARK-32531. The added tests cover cases for maps and arrays of nested structs for different file formats. Eg, [https://github.com/apache/spark/pull/29353] and [https://github.com/apache/spark/pull/29354] add object reuse when reading ORC and Avro files. However, for dynamic data structures like arrays and maps, we do not know just by looking at the schema what the size of the data structure will be so it has to be allocated when reading the data points. The added tests provide coverage so that objects are not accidentally reused when encountering maps and arrays.
AFAIK this is not covered by existing tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org