[jira] [Updated] (PARQUET-343) Caching nulls on group node to improve write performance on wide schema sparse data

["Gabor Szadovszky (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PARQUET-343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gabor Szadovszky updated PARQUET-343:
-------------------------------------
    Fix Version/s: 1.8.2

> Caching nulls on group node to improve write performance on wide schema sparse data
> -----------------------------------------------------------------------------------
>
>                 Key: PARQUET-343
>                 URL: https://issues.apache.org/jira/browse/PARQUET-343
>             Project: Parquet
>          Issue Type: Improvement
>            Reporter: Tianshuo Deng
>            Assignee: Tianshuo Deng
>            Priority: Major
>             Fix For: 1.9.0, 1.8.2
>
>
> or really wide schema with sparse data, If a group node is empty, it could have a huge number of leaves underneath it. Calling writeMull for each leaf every time when it's ancestor group node is null is in-effcient and is bad for data locality in the memory especially when the number of leaves is huge.
> Instead, null can be cached on the group node. Flushing is only triggered when a group node becomes non-null from null. This way, all the cached null values will be flushed to the leaf nodes in a tight loop and improves write performance.
> We tested this approach combined with PARQUET-341 on a really large schema and gave us ~2X improvement on write performance



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)