You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@parquet.apache.org by "Ryan Blue (JIRA)" <ji...@apache.org> on 2015/12/08 23:47:12 UTC

[jira] [Resolved] (PARQUET-382) Add a way to append encoded blocks in ParquetFileWriter

     [ https://issues.apache.org/jira/browse/PARQUET-382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ryan Blue resolved PARQUET-382.
-------------------------------
       Resolution: Fixed
    Fix Version/s: 1.9.0

Merged #278. Thanks for reviewing, Sergio!

> Add a way to append encoded blocks in ParquetFileWriter
> -------------------------------------------------------
>
>                 Key: PARQUET-382
>                 URL: https://issues.apache.org/jira/browse/PARQUET-382
>             Project: Parquet
>          Issue Type: New Feature
>          Components: parquet-mr
>    Affects Versions: 1.8.0
>            Reporter: Ryan Blue
>            Assignee: Ryan Blue
>             Fix For: 1.9.0
>
>
> Concatenating two files together currently requires reading the source files and rewriting the content from scratch. This ends up taking a lot of memory, even if the data is already encoded correctly and blocks just need to be appended and have their metadata updated. Merging two files should be fast and not take much memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)