[jira] [Created] (HDDS-16376) OM is stuck twice when SCM leader has a network issue

["Ivan Andika (Jira)" <ji...@apache.org>]

Ivan Andika created HDDS-16376:
----------------------------------

             Summary: OM is stuck twice when SCM leader has a network issue
                 Key: HDDS-16376
                 URL: https://issues.apache.org/jira/browse/HDDS-16376
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Ivan Andika
            Assignee: Ivan Andika


Another incident.
h2. Problem 1: OM is stuck waiting for SCM port 9863 (block client port) with NIC issue 

The SCM leader NIC encountered an issue where it started dropping all packet. The SCM service was able to quickly elect a new SCM leader but OM is still stuck. It recovered for 20s after 15 minutes and then stuck again for another 30 minutes.

The first 15 minutes timeout is due to the OM waiting for the SCM call to finish hdds.scmclient.rpc.timeout (default 15 minutes). This RPC timeout is used in Hadoop Client.Connection.soTimeout. This soTimeout is used for the Socket.setSoTimeout which will throw SockeTimeoutException if the socket read and write cannot finish on time. 

The main issue is that the NIC issue does not cause the RPC connection to be broken (e.g. by TCP reset) which will trigger a fast IPC failure. The NIC issue only causes the packet to be dropped on an active connection. Therefore, the TCP connection keep retrying sending the packet although it's going to be dropped.

After the 15 minutes timeout is finished, the socket timeout is hit and OM's SCM client failover is triggered and start to the new SCM leader.
h2. Problem 2: OM is stuck again for port 9860 (container client port) after recovering for a while

After recovering for around 20 seconds, OM is stuck again. This time is because the SCM client for the container client port still thinks the problematic SCM was still the leader.

Saw some logs for retying connection repeated every 20s
{code:java}
Retrying connect to server: scm1:9860. Already tried 0 time(s); maxRetries=45 
Retrying connect to server: scm1:9860. Already tried 1 time(s); maxRetries=45
...
Retrying connect to server: scm1:9860. Already tried 44 time(s); maxRetries=45 {code}
The issue happened when the RPC client is trying to setup a connection to the SCM leader but the connection cannot be established due to ConnectTimeoutException (see Client.Connection#setupConnection).When ConnectionTimeoutException is received, the client will wait for the total of ipc.client.connect.timeout (20 s) × ipc.client.connect.max.retries.on.timeouts (45) = 920s (15 minutes 20 seconds)However, this means that the SCM client should have recovered after by 07:39 (07:24 + 15 minutes). However, it seems that the retry cycled again and redo the whole 45 retries again.
{code:java}
08-29 07:53:49.242: Retrying connect to server: scm1:9860. Already tried 44 time(s); maxRetries=45
# cycling back to 0
08-29 07:54:29.282: Retrying connect to server: scm1:9860. Already tried 0 time(s); maxRetries=45 / code placeholder
{code}
Seems the issue is that SCM failover is not atomic, meaning that a IPC client can trigger setupConnection while the SCM is undergoing a failover and trigger another connection timeout retry.
h2. Possible solutions
h3. Reduce the RPC call  timeout

We can check the timeout related configurations
 * {{Connection timeout}}
 ** {{ipc.client.connect.max.retries.on.timeouts}} (default 45)
 ** ipc.client.connect.timeout (default 20s)

 * {{RPC timeout}}
 ** {{hdds.scmclient.rpc.timeout}} (default 15s){{{{}}{}}}{{{{}}{}}}

h3. Make SCM client aware of other client failovers

The current issue is that both container client and block client have their own failover states which can cause 2x timeout. The question is whether it is possible to make one SCM client failover to notify the other SCM client to not send a request to another request.
h3. Fix the SCM failover race

The important race is that SCM failover happens outside Hadoop IPC:
 # Hadoop IPC first exhausts its 45 connect-timeout retries.
 # It returns an exception to the Ozone SCM {{{}RetryProxy{}}}.
 # The SCM retry policy waits 2 seconds.
 # The {{RetryProxy}} then advances to the next configured SCM.

During the short interval between steps 1 and 4, another concurrent OM request can obtain the still-current {{scm1}} proxy and create a new Hadoop IPC connection. Its local count starts at zero and it can independently spend another ~15 minutes retrying.

The counter is declared locally for each {{setupConnection()}} call, so it necessarily resets for a newly created connection.

The client should switch after a completed failover, but it does not atomically prevent already-starting or concurrent requests from beginning a fresh long connect-retry cycle on the old node.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org