You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Weston Pace (Jira)" <ji...@apache.org> on 2021/06/22 02:20:00 UTC
[jira] [Updated] (ARROW-8599) [C++][Parquet] Optional parallel
processing when writing Parquet files
[ https://issues.apache.org/jira/browse/ARROW-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Weston Pace updated ARROW-8599:
-------------------------------
Fix Version/s: (was: 5.0.0)
6.0.0
> [C++][Parquet] Optional parallel processing when writing Parquet files
> ----------------------------------------------------------------------
>
> Key: ARROW-8599
> URL: https://issues.apache.org/jira/browse/ARROW-8599
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Wes McKinney
> Assignee: Weston Pace
> Priority: Major
> Fix For: 6.0.0
>
>
> If we permit encoded columns in row groups to be buffered in memory rather than immediately written out to the {{OutputStream}}, then we can use multiple threads for the encoding / compression of columns. Combined with a separate thread to take the encoded columns and write them out to disk, this should yield substantially improved file write times.
> This could be enabled through an option since it would increase memory use when writing. The memory use can be somewhat constrained by limiting the size of row groups
--
This message was sent by Atlassian Jira
(v8.3.4#803005)