[jira] [Created] (HDFS-18005) Add an option to skip hedged reads when the first-choice replica is local and short-circuit reads are enabled

["Hernan Romer (Jira)" <ji...@apache.org>]

Hernan Romer created HDFS-18005:
-----------------------------------

             Summary: Add an option to skip hedged reads when the first-choice replica is local and short-circuit reads are enabled
                 Key: HDFS-18005
                 URL: https://issues.apache.org/jira/browse/HDFS-18005
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: hdfs-client
            Reporter: Hernan Romer


When hedged reads are enabled, every positional read on a replicated file goes through DFSInputStream#hedgedFetchBlockByteRange, including reads that finish well under the hedging threshold. That path allocates a new heap ByteBuffer for every attempt, hands even the first attempt off to the hedged-read pool, copies the result into the caller's buffer, and so loses zero-copy short-circuit reads into a caller-supplied direct buffer. The separate buffer is required because losing attempts are not interrupted (HDFS-10182). For workloads with high data locality, such as HBase RegionServers co-located with DataNodes, most preads are local short-circuit reads. They rarely benefit from hedging, but they still pay this cost and take up pool threads. Operators therefore have to choose between that overhead on every read and no protection against slow or hung remote DataNodes.

We propose adding a client option, dfs.client.hedged.read.skip.short.circuit, defaulting to false. When it is enabled, pread would use the non-hedged path if the first-choice replica is local, dfs.client.read.shortcircuit is true, and the file is not under construction. All other reads would be hedged as today, and striped reads would be unaffected. To avoid selecting a replica twice, pread could choose the DataNode once and pass it to whichever fetch path it uses. One trade-off to note: a read that skips hedging would not hedge a slow local read, or its retries on other replicas after a local failure. With the option off, behavior would be unchanged.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org