[jira] [Created] (HBASE-27896) Disable hdfs readahead for pread reads

["Bryan Beaudreault (Jira)" <ji...@apache.org>]

Bryan Beaudreault created HBASE-27896:
-----------------------------------------

             Summary: Disable hdfs readahead for pread reads
                 Key: HBASE-27896
                 URL: https://issues.apache.org/jira/browse/HBASE-27896
             Project: HBase
          Issue Type: Improvement
            Reporter: Bryan Beaudreault


In https://issues.apache.org/jira/browse/HBASE-17914, a flag was introduced {{{}hbase.store.reader.no-readahead{}}}. The default is false, so readahead is enabled. This flag is used for creating the default store reader (i.e. the one used by PREAD reads). Stream readers don't use this flag, instead they always pass -1.

When that flag is false, we pass a readahead value of 0 to FSDataInputStream.setReadahead. When the flag is true, we pass -1 which triggers hdfs default behavior. The default behavior is to use a readahead of 4MB.

It seems to me that we don't want readahead for PREAD reads, and especially not such a large readahead. Our default block size is 64kb, which is much smaller than that. A PREAD read is not likely to do sequential IO, so not likely to utilize the cached readahead buffer.

I set no-readahead to true in a few of our clusters and in each case saw a massive reduction in disk IO and thus increase in throughput. I load tested this in a test cluster which does fully random reads of ~300 byte rows on a dataset which is 20x larger than memory. The load test was able to achieve nearly double the throughput.

As a follow-on, we might consider tuning the readahead for STREAM reads. 4mb seems way too big for many common workloads.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)