[jira] [Commented] (HDFS-2891) Some times first DataNode detected as bad when we power off for the second DataNode.

["chackaravarthy (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13243686#comment-13243686 ] 

chackaravarthy commented on HDFS-2891:
--------------------------------------

Hi Uma,

This problem exists in branch-1.0 also.

*This can happen in the following scenario :*
{quote}
1. consider the pipeline as [ DN1 -> DN2 -> DN3 ]
2. create one file and get the stream.
3. write some bytes using that stream and call sync.
4. keep the stream open.
5. Now unplug network / power-off / ethernet down in DN2 machine.
{quote}

*The explanation is as follows :*

*Consider the case, when the caller is not writing any data and dataStreamer & ResponseProcessor threads are running And also DN2 machine ethernet down:*

{quote}
	1. At *time t1* , ResponseProcessor will start reading the ack from DN1 [timeOut is 69 secs ]

	2. But in the DN1 , packetResponder not yet started reading the ack. It will be waiting on ackQueue until one packet arrives.

	3. After time *t1+34.5* secs only, dataStreamer will stream the HEART_BEAT packet to DN1. [if there is no data packet, DataStreamer will send HEART_BEAT packet after waiting for half of the timeout value]

	4. Then only, DataXceiver will receive the packet and will put it in ackQueue in DN side.

	5. At *time t2* , Once the ackQueue is enqueued, the packetResponder will start reading the ack from DN2. [timeOut is 66 secs]

	6. As DN2 machine ethernet is down, packetResponder in DN1 is not getting the reply.

	7. But packetResponder will get timeOut only after *t2+66* secs.

	8. Hence ResponseProcessor is getting socketTimeOutException earlier than PacketResponder.

{quote}

		*t2 - t1 >= 34.5 secs*   [if its greater than 3 secs itself, the reported scenario can happen]

	So, DFSClient is getting SocketTimeOutException before DN1.
	Hence DFSClient is detecting DN1 as bad datanode [which is up] and not detecting DN2 as bad datanode [which is down]


*Reason :*

In DataNode, the PacketResponder will start reading the ack only when it receives one packet [either data or heartbeat packet]
But In DFSClient, the ResponseProcessor will start reading the ack before sending packet.

                
> Some times first DataNode detected as bad when we power off for the second DataNode.
> ------------------------------------------------------------------------------------
>
>                 Key: HDFS-2891
>                 URL: https://issues.apache.org/jira/browse/HDFS-2891
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node, hdfs client
>    Affects Versions: 1.1.0
>            Reporter: Uma Maheswara Rao G
>
> In one of my clusters, observed this situation.
> This issue looks to be due to time out in ResponseProcesser at client side, it is marking first DataNode as bad.
> This happens in 20.2 version. This can be there in branch-1 as well and will check for trunk.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira