[jira] [Commented] (HDFS-12098) Ozone: Datanode is unable to register with scm if scm starts later

["Weiwei Yang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-12098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078213#comment-16078213 ] 

Weiwei Yang commented on HDFS-12098:
------------------------------------

This is because datanode state machine leaks {{VersionEndpointTask}} thread. In the case scm is not yet started,
 more and more {{VersionEndpointTask}} threads keep retrying connection with scm,

{noformat}
INIT - RUNNING 
                 \
                GETVERSION
                       executor.execute(new VersionEndpointTask()) - retry on getVersion ...
                               ... (HB interval)
                       executor.execute(new VersionEndpointTask()) - retry on getVersion ...
                               ... (HB interval)
                       executor.execute(new VersionEndpointTask()) - retry on getVersion ...
                               ...
{noformat}

the version endpoint tasks are launched in HB interval (5s on my env), so every 5s there is a new task submitted; the retry policy for each getVersion call is 10 * 1s = 10s, so every 10s a task can be finished. So every 10s there will be ONE thread leak.

When scm is up, all pending tasks will be able to connect to scm and getVersion call returns, so each of them will count the state to next, since the state is shared in {{EndpointStateMachine}}, it increments more than 1 so when I review the state changes, it looks like below

{noformat}
REGISTER
HEARTBEAT
SHUTDOWN
SHUTDOWN
SHUTDOWN
... 
{noformat}

> Ozone: Datanode is unable to register with scm if scm starts later
> ------------------------------------------------------------------
>
>                 Key: HDFS-12098
>                 URL: https://issues.apache.org/jira/browse/HDFS-12098
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, ozone, scm
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>            Priority: Critical
>
> Reproducing steps
> # Start datanode
> # Wait and see datanode state, it has connection issues, this is expected
> # Start SCM, expecting datanode could connect to the scm and the state machine could transit to RUNNING. However in actual, its state transits to SHUTDOWN, datanode enters chill mode.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org