[jira] [Commented] (PARQUET-1634) [C++] Factor out data/dictionary page writes to allow for page buffering

["Wes McKinney (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902357#comment-16902357 ] 

Wes McKinney commented on PARQUET-1634:
---------------------------------------

To assist with this it would make sense to first make sure we have a mock "high latency" filesystem for testing / benchmarking

> [C++] Factor out data/dictionary page writes to allow for page buffering 
> -------------------------------------------------------------------------
>
>                 Key: PARQUET-1634
>                 URL: https://issues.apache.org/jira/browse/PARQUET-1634
>             Project: Parquet
>          Issue Type: Improvement
>          Components: parquet-cpp
>            Reporter: Wes McKinney
>            Priority: Major
>             Fix For: cpp-1.6.0
>
>
> Logic that eagerly writes out data pages is hard-coded into the column writer implementation
> https://github.com/apache/arrow/blob/master/cpp/src/parquet/column_writer.cc#L565
> For higher-latency file systems like Amazon S3, it makes more sense to buffer pages in memory and write them in larger batches (and preferably asynchronously). We should refactor this logic so we have the ability to choose rather than have the behavior hard-coded



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)