You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Sean Bigdatafun <se...@gmail.com> on 2011/01/03 09:49:34 UTC
How does HDFS handle a failed Datanode during write?
I'd like to understand how HDFS handle Datanode failure gracefully. Let's
suppose a replication factor of 3 is used in HDFS for this discussion.
After 'DataStreamer' receives a list of Datanodes A, B, C for a block, it
starts pulling data packets off the 'data queue' and putting it onto 'ack
queue' after sending them off the wire to those Datanodes (using a pipeline
mechansim Client -> A -> B -> C). If the Datanode B crashes during the
writing, why the client need to put the data packets in the 'ack queue'
back to the 'data queue'? (how can the client guarantee the order of resent
packet on Datanode A after all?)
I guess I have not fully understood the write failure handling mechanism
yet. Can someone give a detailed explanation?
Thanks,
--
--Sean
--
--Sean
Re: How does HDFS handle a failed Datanode during write?
Posted by Thanh Do <th...@cs.wisc.edu>.
some details can be found here
appendDesign3.pdf<https://issues.apache.org/jira/secure/attachment/12445209/appendDesign3.pdf>
<https://issues.apache.org/jira/secure/attachment/12445209/appendDesign3.pdf>
Thanh
On Mon, Jan 3, 2011 at 2:49 AM, Sean Bigdatafun
<se...@gmail.com>wrote:
> I'd like to understand how HDFS handle Datanode failure gracefully. Let's
> suppose a replication factor of 3 is used in HDFS for this discussion.
>
>
> After 'DataStreamer' receives a list of Datanodes A, B, C for a block, it
> starts pulling data packets off the 'data queue' and putting it onto 'ack
> queue' after sending them off the wire to those Datanodes (using a pipeline
> mechansim Client -> A -> B -> C). If the Datanode B crashes during the
> writing, why the client need to put the data packets in the 'ack queue'
> back to the 'data queue'? (how can the client guarantee the order of resent
> packet on Datanode A after all?)
> I guess I have not fully understood the write failure handling mechanism
> yet. Can someone give a detailed explanation?
>
> Thanks,
> --
> --Sean
>
>
>
>
>
> --
> --Sean
>
>
>
Re: How does HDFS handle a failed Datanode during write?
Posted by Dhruba Borthakur <dh...@gmail.com>.
each packet has an offset in the file that it is supposed to be written to.
So, there is no hard in resending the same packet twice, the receiving
datanode would always write this packet to the correct offset in the
destination file.
If B crashes during the write, the client does not know whether the write
was successful at all the replicas. So, the client bumps up the generation
stamp of the block and then *resends* all the pending packets to all the
datanodes.
thanks,
dhruba
On Mon, Jan 3, 2011 at 12:49 AM, Sean Bigdatafun
<se...@gmail.com>wrote:
> I'd like to understand how HDFS handle Datanode failure gracefully. Let's
> suppose a replication factor of 3 is used in HDFS for this discussion.
>
>
> After 'DataStreamer' receives a list of Datanodes A, B, C for a block, it
> starts pulling data packets off the 'data queue' and putting it onto 'ack
> queue' after sending them off the wire to those Datanodes (using a pipeline
> mechansim Client -> A -> B -> C). If the Datanode B crashes during the
> writing, why the client need to put the data packets in the 'ack queue'
> back to the 'data queue'? (how can the client guarantee the order of resent
> packet on Datanode A after all?)
> I guess I have not fully understood the write failure handling mechanism
> yet. Can someone give a detailed explanation?
>
> Thanks,
> --
> --Sean
>
>
>
>
>
> --
> --Sean
>
--
Connect to me at http://www.facebook.com/dhruba
Re: How does HDFS handle a failed Datanode during write?
Posted by Dhruba Borthakur <dh...@gmail.com>.
each packet has an offset in the file that it is supposed to be written to.
So, there is no hard in resending the same packet twice, the receiving
datanode would always write this packet to the correct offset in the
destination file.
If B crashes during the write, the client does not know whether the write
was successful at all the replicas. So, the client bumps up the generation
stamp of the block and then *resends* all the pending packets to all the
datanodes.
thanks,
dhruba
On Mon, Jan 3, 2011 at 12:49 AM, Sean Bigdatafun
<se...@gmail.com>wrote:
> I'd like to understand how HDFS handle Datanode failure gracefully. Let's
> suppose a replication factor of 3 is used in HDFS for this discussion.
>
>
> After 'DataStreamer' receives a list of Datanodes A, B, C for a block, it
> starts pulling data packets off the 'data queue' and putting it onto 'ack
> queue' after sending them off the wire to those Datanodes (using a pipeline
> mechansim Client -> A -> B -> C). If the Datanode B crashes during the
> writing, why the client need to put the data packets in the 'ack queue'
> back to the 'data queue'? (how can the client guarantee the order of resent
> packet on Datanode A after all?)
> I guess I have not fully understood the write failure handling mechanism
> yet. Can someone give a detailed explanation?
>
> Thanks,
> --
> --Sean
>
>
>
>
>
> --
> --Sean
>
--
Connect to me at http://www.facebook.com/dhruba
Re: How does HDFS handle a failed Datanode during write?
Posted by Thanh Do <th...@cs.wisc.edu>.
some details can be found here
appendDesign3.pdf<https://issues.apache.org/jira/secure/attachment/12445209/appendDesign3.pdf>
<https://issues.apache.org/jira/secure/attachment/12445209/appendDesign3.pdf>
Thanh
On Mon, Jan 3, 2011 at 2:49 AM, Sean Bigdatafun
<se...@gmail.com>wrote:
> I'd like to understand how HDFS handle Datanode failure gracefully. Let's
> suppose a replication factor of 3 is used in HDFS for this discussion.
>
>
> After 'DataStreamer' receives a list of Datanodes A, B, C for a block, it
> starts pulling data packets off the 'data queue' and putting it onto 'ack
> queue' after sending them off the wire to those Datanodes (using a pipeline
> mechansim Client -> A -> B -> C). If the Datanode B crashes during the
> writing, why the client need to put the data packets in the 'ack queue'
> back to the 'data queue'? (how can the client guarantee the order of resent
> packet on Datanode A after all?)
> I guess I have not fully understood the write failure handling mechanism
> yet. Can someone give a detailed explanation?
>
> Thanks,
> --
> --Sean
>
>
>
>
>
> --
> --Sean
>
>
>