[jira] [Created] (HDDS-17322) Datanode heartbeat to an unreachable SCM or Recon can block about 15 minutes on connect timeout retries

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17322:
---------------------------------

             Summary: Datanode heartbeat to an unreachable SCM or Recon can block about 15 minutes on connect timeout retries
                 Key: HDDS-17322
                 URL: https://issues.apache.org/jira/browse/HDDS-17322
             Project: Apache Ozone
          Issue Type: Bug
          Components: Ozone Datanode
            Reporter: Siyao Meng
            Assignee: Siyao Meng


{{SCMConnectionManager}} creates the Datanode's RPC proxies to SCM and Recon with {{retryUpToMaximumCountWithFixedSleep(15, 1s)}}. That policy only covers connection failures such as "connection refused". A connect timeout (the host drops packets: a stopped VM, a firewall DROP rule, a stale Kubernetes pod IP) is handled separately by the forked {{ipc_.Client}}, which retries it {{ipc.client.connect.max.retries.on.timeouts}} (45) times with {{ipc.client.connect.timeout}} (20s) each. By calculation that is about 15 minutes per call. Neither key is overridden for these proxies.

{{RunningDatanodeState}} waits for an endpoint task for at most one heartbeat interval but does not cancel it, and each endpoint runs its tasks on a single thread with an unbounded queue (see HDDS-17269). So while an SCM is unreachable this way, one heartbeat task is queued per interval and each one blocks for about 15 minutes. Each missed heartbeat takes about 15 minutes instead of one interval, so re-resolving the SCM address after {{hdds.heartbeat.address.refresh.missed.count.threshold}} missed heartbeats (when enabled) is delayed by the same factor.

Proposed fix: create these proxies with {{ipc.client.connect.max.retries.on.timeouts}} set to 0 unless it is set explicitly, as HDDS-17310 does for the OM and SCM client failover proxies, and let the Datanode's own retry policy and heartbeat cycle handle the retries.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org