You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@parquet.apache.org by "Gang Wu (Jira)" <ji...@apache.org> on 2023/07/06 14:20:00 UTC

[jira] [Resolved] (PARQUET-2316) Allow partial prebuffer in parquet FileReader

     [ https://issues.apache.org/jira/browse/PARQUET-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gang Wu resolved PARQUET-2316.
------------------------------
    Resolution: Fixed

Issue resolved by pull request 36192
https://github.com/apache/arrow/pull/36192

> Allow partial prebuffer in parquet FileReader
> ---------------------------------------------
>
>                 Key: PARQUET-2316
>                 URL: https://issues.apache.org/jira/browse/PARQUET-2316
>             Project: Parquet
>          Issue Type: Improvement
>          Components: parquet-cpp
>            Reporter: Jinpeng Zhou
>            Assignee: Jinpeng Zhou
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: cpp-13.0.0
>
>          Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> The current FileReader can only work in  one of the two modes, coalescing (when Prebuffer is called) and non-coalescing (when Prefufer is not called), due to the if statement [here|https://github.com/apache/arrow/blob/main/cpp/src/parquet/file_reader.cc#L203]
> Since Prebuffer is basically caching all specified column chunks, it would raise concerns on memory usage for systems with tight memory budget. In such scenarios, one may want to Prebuffer some small chunks while being able to read the rest chunks using  BufferedInputStream. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)