[jira] [Commented] (HADOOP-18854) add options to disable range merging of vectored io

["Mukund Thakur (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-18854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825443#comment-17825443 ] 

Mukund Thakur commented on HADOOP-18854:
----------------------------------------

There is already an option to disable merging 
{code:java}
<property>
   <name>fs.s3a.vectored.read.max.merged.size</name>
   <value>1M</value>
   <description>
      What is the largest merged read size in bytes such
      that we group ranges together during vectored read.
      Setting this value to 0 will disable merging of ranges.
   </description>
<property>
 {code}

> add options to disable range merging of vectored io
> ---------------------------------------------------
>
>                 Key: HADOOP-18854
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18854
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs, fs/s3
>    Affects Versions: 3.3.5, 3.3.6
>            Reporter: Steve Loughran
>            Priority: Major
>
> I'm seeing test failures in my PARQUET-2171 pr because assertions about the #of bytes read isn't holding -small files are being read and the vector range merging is pulling in the whole file.
> ```
> [ERROR]   TestInputOutputFormat.testReadWriteWithCounter:338 bytestotal != bytesread expected:<5510> but was:<11020>
> ```
> I think for parquet i will add an option to disable vector io, but really the filesystems which support it should allow for merging to be disabled



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org