[jira] [Commented] (PARQUET-1626) [C++] Ability to concat parquet files

["David Lee (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16901509#comment-16901509 ] 

David Lee commented on PARQUET-1626:
------------------------------------

I'm appending RowGroups using pyarrow today.

Open a new parquet file

For each row group in File 1:

    read row group. write row group to new file

For each row group in File 2:

    read row group. write row group to new file

Close new parquet file

No need to mess with metadata since all those stats are saved at a row group level.

I usually generate parquet files which are 30 to 40 megs each and I merge them afterwards to match the HDFS blocksize.

> [C++] Ability to concat parquet files 
> --------------------------------------
>
>                 Key: PARQUET-1626
>                 URL: https://issues.apache.org/jira/browse/PARQUET-1626
>             Project: Parquet
>          Issue Type: New Feature
>          Components: parquet-cpp
>    Affects Versions: cpp-1.3.1
>            Reporter: nileema shingte
>            Priority: Major
>              Labels: features
>
> Ability to concat the parquet files is something we've wanted for some time too. When we generate parquet files partitioned by an expression, we often end up with tiny files and would like to add a post-processing step to concat these files together.
> Is there a plan to add this ability to the library any time soon? 
> If not, it would be great if someone can provide a somewhat detailed pseudocode (expanding on what [~xhochy] mentioned in the comment in PARQUET-1022) as a guideline for conditions/scenarios that need to be handled with extra care, so we can contribute this as a PR. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)