You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "David Li (Jira)" <ji...@apache.org> on 2022/10/20 18:27:00 UTC
[jira] [Commented] (ARROW-18113) Implement a read range process without caching
[ https://issues.apache.org/jira/browse/ARROW-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17621308#comment-17621308 ]
David Li commented on ARROW-18113:
----------------------------------
I'd like to raise my comments in ARROW-17913 and ARROW-17917 as well, especially https://issues.apache.org/jira/browse/ARROW-17913?focusedCommentId=17614155&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17614155 . Note Weston offers suggestions there too about how we might handle this.
The API would probably be an extension of RandomAccessFile::ReadAsync. The file system would come into play by returning a subclass of RAF that overrides the new method and does coalescing appropriate to the underlying device
> Implement a read range process without caching
> ----------------------------------------------
>
> Key: ARROW-18113
> URL: https://issues.apache.org/jira/browse/ARROW-18113
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Percy Camilo Triveño Aucahuasi
> Assignee: Percy Camilo Triveño Aucahuasi
> Priority: Major
>
> The current [ReadRangeCache|https://github.com/apache/arrow/blob/e06e98db356e602212019cfbae83fd3d5347292d/cpp/src/arrow/io/caching.h#L100] is mixing caching with coalescing and making difficult to implement readers capable to really perform concurrent reads on coalesced data (see this [github comment|https://github.com/apache/arrow/pull/14226#discussion_r999334979] for additional context); for instance, right now the prebuffering feature of those readers cannot handle concurrent invocations.
> The goal for this ticket is to implement a similar component to ReadRangeCache for performing non-cache reads (doing only the coalescing part instead). So, once we have that new capability, we can port the parquet and IPC readers to this new component and keep improving the reading process (that would be part of other set of follow-up tickets). Similar ideas were mentioned here https://issues.apache.org/jira/browse/ARROW-17599
> Maybe a good place to implement this new capability is inside the file system abstraction (as part of a dedicated method to read coalesced data) and where the abstract file system can provide a default implementation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)