You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Muhammad Samir Khan (Jira)" <ji...@apache.org> on 2020/08/04 20:00:12 UTC
[jira] [Updated] (SPARK-32532) Improve ORC read/write performance
on nested structs and array of structs
[ https://issues.apache.org/jira/browse/SPARK-32532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Muhammad Samir Khan updated SPARK-32532:
----------------------------------------
Description:
Have some improvements for ORC file format to reduce time taken when reading/writing nested/array'd structs. Using benchmarks in [SPARK-32531] was able to improve performance as follows (measurements in seconds):
Read:
Nested Structs: 184 -> 44
Array of Struct: 66 -> 15
Write
Nested Structs: 543 -> 39
Array of Struct: 330 -> 37
Will be putting up the PR soon with the changes.
was:
Have some improvements for ORC file format to reduce time taken when reading/writing nested/array'd structs. Using benchmarks in [SPARK-32071] was able to improve performance as follows (measurements in seconds):
Read:
Nested Structs: 184 -> 44
Array of Struct: 66 -> 15
Write
Nested Structs: 543 -> 39
Array of Struct: 330 -> 37
Will be putting up the PR soon with the changes.
> Improve ORC read/write performance on nested structs and array of structs
> -------------------------------------------------------------------------
>
> Key: SPARK-32532
> URL: https://issues.apache.org/jira/browse/SPARK-32532
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Muhammad Samir Khan
> Priority: Major
>
> Have some improvements for ORC file format to reduce time taken when reading/writing nested/array'd structs. Using benchmarks in [SPARK-32531] was able to improve performance as follows (measurements in seconds):
> Read:
> Nested Structs: 184 -> 44
> Array of Struct: 66 -> 15
> Write
> Nested Structs: 543 -> 39
> Array of Struct: 330 -> 37
> Will be putting up the PR soon with the changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org