[jira] [Updated] (ARROW-8599) [C++][Parquet] Optional parallel processing when writing Parquet files

["Wes McKinney (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wes McKinney updated ARROW-8599:
--------------------------------
    Fix Version/s:     (was: 1.0.0)
                   2.0.0

> [C++][Parquet] Optional parallel processing when writing Parquet files
> ----------------------------------------------------------------------
>
>                 Key: ARROW-8599
>                 URL: https://issues.apache.org/jira/browse/ARROW-8599
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++
>            Reporter: Wes McKinney
>            Priority: Major
>             Fix For: 2.0.0
>
>
> If we permit encoded columns in row groups to be buffered in memory rather than immediately written out to the {{OutputStream}}, then we can use multiple threads for the encoding / compression of columns. Combined with a separate thread to take the encoded columns and write them out to disk, this should yield substantially improved file write times.
> This could be enabled through an option since it would increase memory use when writing. The memory use can be somewhat constrained by limiting the size of row groups



--
This message was sent by Atlassian Jira
(v8.3.4#803005)