You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Apache Arrow JIRA Bot (Jira)" <ji...@apache.org> on 2022/11/09 17:52:00 UTC
[jira] [Assigned] (ARROW-17380) [C++] [Python] Tag record batches with start_byte and end_byte infromation
[ https://issues.apache.org/jira/browse/ARROW-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Arrow JIRA Bot reassigned ARROW-17380:
---------------------------------------------
Assignee: (was: Ziheng Wang)
> [C++] [Python] Tag record batches with start_byte and end_byte infromation
> --------------------------------------------------------------------------
>
> Key: ARROW-17380
> URL: https://issues.apache.org/jira/browse/ARROW-17380
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, Python
> Reporter: Ziheng Wang
> Priority: Major
>
> It might be desirable for a record batch to have information of where it came from in the source dataset. This can be used for a few purposes:
> * Rereading a particular record batch without rereading the entire fragment
> * Easily tracking progress of how much a particular (file) dataset has been consumed.
> It could also be useful for debugging if a record batch resulted in an error downstream.
> The plan is to add some attribute like this here: [https://github.com/apache/arrow/blob/master/cpp/src/arrow/dataset/scanner.cc#L923] that will be tagged on to the record batch by the Scanner as it is being generated.
> This is useful for file based formats like CSV. In Parquet this is less necessary since record batches (usually) correspond to row groups and row group ids can be used to serve this function.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)