[jira] [Updated] (HDDS-17327) Topology-aware read tries an unresponsive nearest datanode first for every ReadChunk

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17327:
------------------------------
    Epic Link: HDDS-17321

> Topology-aware read tries an unresponsive nearest datanode first for every ReadChunk
> ------------------------------------------------------------------------------------
>
>                 Key: HDDS-17327
>                 URL: https://issues.apache.org/jira/browse/HDDS-17327
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: Ozone Client
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>
> With {{ozone.network.topology.aware.read}}=true (the default), {{XceiverClientGrpc.sortDatanodes}} orders the replicas by distance for every ReadChunk, and {{sendCommandWithRetry}} tries them in that order. Nothing remembers that a datanode just failed: {{XceiverClientGrpc}} keeps no failed datanodes, and the exclusion in {{ContainerProtocolCalls.tryEachDatanode}} only lasts for one call. If the nearest datanode hangs (e.g. a stuck disk or a long GC pause) instead of refusing connections, every ReadChunk waits {{ozone.client.read.timeout}} (30s) before the next replica answers. A ReadChunk is sent for each read call within a chunk, so by calculation reading a 256MB block with 4MB reads takes up to 64 × 30s, about 32 minutes, longer.
> HDDS-16419 removes the nested N × N failover, but keeps the same order for each call.
> Proposed fix: keep a set of datanodes that failed recently, with an expiry like {{ozone.client.exclude.nodes.expiry.time}} on the write side, and try them last.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org