[jira] [Commented] (ORC-262) Support async prefetch in Orc reader

["Xinyu Zeng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ORC-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17567778#comment-17567778 ] 

Xinyu Zeng commented on ORC-262:
--------------------------------

Any update on this issue? Does ORC cpp reader have async io now?

> Support async prefetch in Orc reader
> ------------------------------------
>
>                 Key: ORC-262
>                 URL: https://issues.apache.org/jira/browse/ORC-262
>             Project: ORC
>          Issue Type: Improvement
>          Components: C++
>            Reporter: Xiening Dai
>            Priority: Major
>
> Currently RowReader::next() method reads a batch of rows and return them to be processed by runtime. The function call is synchronized, meaning that the execution thread is blocked while reader is loading data from disk. We could potentially parallelize the execution and data loading through async prefetch using logic described as below.
> In SeekableFileInputStream::Next(), we firstly check if the requested data block is already prefetched, if yes, we simply return the buffer to the caller, otherwise we issue a sync call to read data from file stream. No matter how we load the requested data block, we always issue another async call to prefetch the next block within current stream. 
> Additionally orc::InputStream will need a new method that does the async read for a given offset and length.
> According to our experiment, async prefetch can significantly reduce the IO wait time on a heavy loaded distributed file system. By carefully choosing the prefetch data block size, we can maximize the parallelization of runtime execution and data loading, and achieve a relatively high cache hit rate (~85%).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)