[jira] [Resolved] (HADOOP-19229) Vector IO: have a max distance between ranges to read

["Steve Loughran (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-19229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran resolved HADOOP-19229.
-------------------------------------
    Release Note: This is exactly what `"fs.s3a.vectored.read.min.seek.size" does. We set it to 4K; maybe we should review it. The facebook Velox paper says that 20kB is better for cloud storage
      Resolution: Fixed

> Vector IO: have a max distance between ranges to read
> -----------------------------------------------------
>
>                 Key: HADOOP-19229
>                 URL: https://issues.apache.org/jira/browse/HADOOP-19229
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.4.1
>            Reporter: Steve Loughran
>            Priority: Major
>
> vector iO has a max size to coalesce ranges, but it also needs a maximum gap between ranges to justify the merge. Right now we could have a read where two vectors of size 8 bytes can be merged with a 1 MB gap between them -and that's wasteful. 
> We could also consider an "efficiency" metric which looks at the ratio of bytes-read to bytes-discarded. Not sure what we'd do with it, but we could track it as an IOStat



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org