You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2021/01/06 10:15:01 UTC
[jira] [Assigned] (ARROW-7800) [Python] Expose GetRecordBatchReader
API in PyArrow
[ https://issues.apache.org/jira/browse/ARROW-7800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joris Van den Bossche reassigned ARROW-7800:
--------------------------------------------
Assignee: Joris Van den Bossche
> [Python] Expose GetRecordBatchReader API in PyArrow
> ---------------------------------------------------
>
> Key: ARROW-7800
> URL: https://issues.apache.org/jira/browse/ARROW-7800
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Reporter: Rollo Konig-brock
> Assignee: Joris Van den Bossche
> Priority: Minor
> Labels: features, pull-request-available
> Fix For: 3.0.0
>
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> The GetRecordBatchReader API is really useful for streaming ParquetFiles with lots of RLE.
> I propose exposing this API in PyArrow in the following manner:
> {code}
> file_ = ParquetFile('file/path.parquet')
> for batch in file_.get_batches(batch_size=100):
> pass
> {code}
> (If anyone has any better ideas hit me up, I'm not 100% sold on exposing it this way.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)