[jira] [Commented] (HDFS-11334) [SPS]: NN switch and rescheduling movements can lead to have more than one coordinator for same file blocks

["Rakesh R (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-11334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15925407#comment-15925407 ] 

Rakesh R commented on HDFS-11334:
---------------------------------

[~umamaheswararao],
Apart from the NN switch and DN heartbeat expiry, there is another interesting case to be handled. Am adding here for discussion:

*Scenario:* Say, there are 3 files {{File1, File2, File3}} given for SPS at different time stamp, {{T1, T2, T3}}. Assume, all three block movements scheduled to DN1 coordinator at three different time stamp {{T1’, T2’, T3’}}. Again assume DN1’s heartbeat missing only for few cycles(less than expiry time). On the other side, consider a case where only {{File1}} becomes attempted timed out during DN's heartbeat missing period and SPS would reassign to a new DN2 coordinator for block movement. Now, DN1 rejoins(before expiry) and reporting block movement results of {{File1}}.

In this case, only {{File1}} got attempted timed out and reassigned to a new coordinator DN2 node. But all the other files {{File2, File3}} are still handled by DN1 which is in progress and he would be able to track till completion as he is not an expired datanode. So the challenge is, to build a mechanism to tell coordinator datanode to drop only the timedout {{trackIds}}. Do you think to introduce {{trackId + genstamp}} based logic to handle corner cases like this ?


> [SPS]: NN switch and rescheduling movements can lead to have more than one coordinator for same file blocks
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-11334
>                 URL: https://issues.apache.org/jira/browse/HDFS-11334
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>    Affects Versions: HDFS-10285
>            Reporter: Uma Maheswara Rao G
>            Assignee: Rakesh R
>
> I am summarizing the scenarios here what Rakesh and me discussed offline:
> Here we need to handle couple of cases:
> # NN switch - it will freshly start scheduling for all files.
>        At this time, old co-ordinators may continue movement work and send results back. This could confuse NN SPS that which result is right one.
>   *NEED TO HANDLE*
> # DN disconnected for heartbeat expiry - If DN disconnected for long time(more than heartbeat expiry), NN will remove this nodes. After SPS Monitor time out, it may retry for files which were scheduled to that DN, by finding new co-ordinator. But if it reconnects back after NN reschedules, it may lead to get different results from deferent co-ordinators.
> *NEED TO HANDLE*
> # NN Restart- Should be same as point 1
> # DN disconnect - here When DN disconnected simply and reconnected immediately (before heartbeat expiry), there should not any issues
> *NEED NOT HANDLE*, but can think of more scenarios if any thing missing
> # DN Restart- If DN restarted, DN can not send any results as it will loose everything. After NN SPS Monitor timeout, it will retry.
> *NEED NOT HANDLE*, but can think of more scenarios if any thing missing



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org