[jira] [Work logged] (HADOOP-11867) Add a high-performance vectored read API.

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-11867?focusedWorklogId=718557&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-718557 ]

ASF GitHub Bot logged work on HADOOP-11867:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 01/Feb/22 10:51
            Start Date: 01/Feb/22 10:51
    Worklog Time Spent: 10m 
      Work Description: steveloughran commented on pull request #3904:
URL: https://github.com/apache/hadoop/pull/3904#issuecomment-1026711104


   +1 to merge into this branch as feature complete, now we follow up with the final integration work.
   
   * remove the extended timeout and lets see if things are good. if not, the change can go in as a single patch into trunk
   * this pr can/should still be rebased as relevant changes go in, it just needs to be co-ordinated with anyone else with the branch checked out
   * but i do want the patches to go in as independent commits for better tracing, with "part of HADOOP-11687" in each patch so a git log --grep will find them all
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 718557)
    Time Spent: 12h 20m  (was: 12h 10m)

> Add a high-performance vectored read API.
> -----------------------------------------
>
>                 Key: HADOOP-11867
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11867
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs, fs/azure, fs/s3, hdfs-client
>    Affects Versions: 3.0.0
>            Reporter: Gopal Vijayaraghavan
>            Assignee: Mukund Thakur
>            Priority: Major
>              Labels: performance, pull-request-available
>          Time Spent: 12h 20m
>  Remaining Estimate: 0h
>
> The most significant way to read from a filesystem in an efficient way is to let the FileSystem implementation handle the seek behaviour underneath the API to be the most efficient as possible.
> A better approach to the seek problem is to provide a sequence of read locations as part of a single call, while letting the system schedule/plan the reads ahead of time.
> This is exceedingly useful for seek-heavy readers on HDFS, since this allows for potentially optimizing away the seek-gaps within the FSDataInputStream implementation.
> For seek+read systems with even more latency than locally-attached disks, something like a {{readFully(long[] offsets, ByteBuffer[] chunks)}} would take of the seeks internally while reading chunk.remaining() bytes into each chunk (which may be {{slice()}}ed off a bigger buffer).
> The base implementation can stub in this as a sequence of seeks + read() into ByteBuffers, without forcing each FS implementation to override this in any way.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org