You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Antoine Pitrou (Jira)" <ji...@apache.org> on 2021/05/04 14:44:00 UTC
[jira] [Resolved] (ARROW-12522) [C++] Implement asynchronous/"lazy"
variants of ReadRangeCache
[ https://issues.apache.org/jira/browse/ARROW-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Antoine Pitrou resolved ARROW-12522.
------------------------------------
Resolution: Fixed
Issue resolved by pull request 10145
[https://github.com/apache/arrow/pull/10145]
> [C++] Implement asynchronous/"lazy" variants of ReadRangeCache
> --------------------------------------------------------------
>
> Key: ARROW-12522
> URL: https://issues.apache.org/jira/browse/ARROW-12522
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: David Li
> Assignee: David Li
> Priority: Major
> Labels: pull-request-available
> Fix For: 5.0.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Currently ReadRangeCache performs both readahead and coalescing. Also, it exposes primarily a blocking API. Two improvements would be useful for implementing async-generator versions of file readers:
> * A method to get a Future<> for a set of read ranges, so that you can asynchronously wait for ranges to be read instead of attempting to read and getting blocked
> * A way to make the cache not perform readahead, so that data is fetched only when requested. (Then, consumers could handle readahead by making multiple requests to the cache.)
> The cache would still act as an actual cache and would still coalesce. (A further improvement might be to allow discarding cache entries. For the purpose of getting AsyncGenerator<RecordBatch>, we don't need a range more than once, so the cache is just wasting memory.)
> This makes it straightforward to adapt synchronous readers into asynchronous ones so long as you know the read ranges up front; you can then cache all the ranges, call WaitFor<>, then hand the buffer to the existing synchronous reader.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)