[jira] [Created] (HDDS-2679) Ratis ring creation might be failed with async pipeline creation

["Marton Elek (Jira)" <ji...@apache.org>]

Marton Elek created HDDS-2679:
---------------------------------

             Summary: Ratis ring creation might be failed with async pipeline creation 
                 Key: HDDS-2679
                 URL: https://issues.apache.org/jira/browse/HDDS-2679
             Project: Hadoop Distributed Data Store
          Issue Type: Improvement
          Components: Ozone Datanode, SCM
            Reporter: Marton Elek


The problem introduced with async pipeline creation:

 
 # Let's say the SCM got registration from three datanodes.
 # A Ratis/THREE pipeline will be created on SCM
 # With the next HB Datanode1(DN1) will receive the CreatePipeline command
 # Datanode1 will start the Ratis server which tries to get votes from DN2 and DN3
 # If DN2 has not yet received the CreatePipeline command (which has high chance with 30sec HB) it will refuse to vote to DN1
 # DN1 will request a  pipeline close from the SCM as there are no votes from DN2 and DN3
 # Pipeline is closed on SCM side, but in the mean time DN2 (finally) receives the pipeline creation command and tries to get votes, but DN1 has a newer group/pipeline id.
 # And so on

If we are lucky enough after a while all DN will receive the container creation at more or less the same time, but if not, SCM couldn't create an Open Ratis

 

Possible solutions:
 * At the very beginning datanode can trust in the peers and learn the group id (but it doesn't cover the case when one pipeline has been closed on DN1 *and* a new pipeline is created but DN2 still has the old pipeline).
 * We can use bidirectional GRPC streaming for datanode scm communication (which is a good idea anyway to make the communication faster). It makes the communication faster but the problem is still there if there is a network blip between scm and DN1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org