[jira] [Reopened] (HADOOP-11867) Add a high-performance vectored read API.

["Mukund Thakur (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-11867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mukund Thakur reopened HADOOP-11867:
------------------------------------

> Add a high-performance vectored read API.
> -----------------------------------------
>
>                 Key: HADOOP-11867
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11867
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs, fs/azure, fs/s3, hdfs-client
>    Affects Versions: 3.0.0
>            Reporter: Gopal Vijayaraghavan
>            Assignee: Mukund Thakur
>            Priority: Major
>              Labels: performance, pull-request-available
>             Fix For: 3.3.5
>
>          Time Spent: 13h
>  Remaining Estimate: 0h
>
> The most significant way to read from a filesystem in an efficient way is to let the FileSystem implementation handle the seek behaviour underneath the API to be the most efficient as possible.
> A better approach to the seek problem is to provide a sequence of read locations as part of a single call, while letting the system schedule/plan the reads ahead of time.
> This is exceedingly useful for seek-heavy readers on HDFS, since this allows for potentially optimizing away the seek-gaps within the FSDataInputStream implementation.
> For seek+read systems with even more latency than locally-attached disks, something like a {{readFully(long[] offsets, ByteBuffer[] chunks)}} would take of the seeks internally while reading chunk.remaining() bytes into each chunk (which may be {{slice()}}ed off a bigger buffer).
> The base implementation can stub in this as a sequence of seeks + read() into ByteBuffers, without forcing each FS implementation to override this in any way.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org