You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Muhammad Samir Khan (Jira)" <ji...@apache.org> on 2020/08/04 20:00:00 UTC

[jira] [Created] (SPARK-32532) Improve ORC read/write performance on nested structs and array of structs

Muhammad Samir Khan created SPARK-32532:
-------------------------------------------

             Summary: Improve ORC read/write performance on nested structs and array of structs
                 Key: SPARK-32532
                 URL: https://issues.apache.org/jira/browse/SPARK-32532
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: Muhammad Samir Khan


Have some improvements for ORC file format to reduce time taken when reading/writing nested/array'd structs. Using benchmarks in [SPARK-32071] was able to improve performance as follows (measurements in seconds):

Read:
Nested Structs: 184 -> 44
Array of Struct: 66 -> 15

Write
Nested Structs: 543 -> 39
Array of Struct: 330 -> 37

Will be putting up the PR soon with the changes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org