You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Arthur.hk.chan@gmail.com" <ar...@gmail.com> on 2014/12/17 07:05:12 UTC
How will Hadoop handle it when a datanode server with total hardware failure?
Hi,
If each of my datanode servers has 8 hard disks (a 10-node cluster) and I use the default replication factor of 3, how will Hadoop handle it when a datanode with total hardware failure suddenly?
Regards
Arthur
Re: How will Hadoop handle it when a datanode server with total hardware failure?
Posted by zhangyujian1984 <zh...@gmail.com>.
i think hadoop treats a node as a minimal unit, which means a internal failure will be regards as a node failure.
datanode uses heartbeat to keep block alive, if any block failure was detected by namenode, it will be recovered
in another datanode.
Yujian Zhang
Southeast University, Nanjing, China
From: Arthur.hk.chan@gmail.com
Date: 2014-12-17 14:05
To: <us...@hadoop.apache.org>
CC: Arthur.hk.chan@gmail.com
Subject: How will Hadoop handle it when a datanode server with total hardware failure?
Hi,
If each of my datanode servers has 8 hard disks (a 10-node cluster) and I use the default replication factor of 3, how will Hadoop handle it when a datanode with total hardware failure suddenly?
Regards
Arthur
Re: How will Hadoop handle it when a datanode server with total hardware failure?
Posted by zhangyujian1984 <zh...@gmail.com>.
i think hadoop treats a node as a minimal unit, which means a internal failure will be regards as a node failure.
datanode uses heartbeat to keep block alive, if any block failure was detected by namenode, it will be recovered
in another datanode.
Yujian Zhang
Southeast University, Nanjing, China
From: Arthur.hk.chan@gmail.com
Date: 2014-12-17 14:05
To: <us...@hadoop.apache.org>
CC: Arthur.hk.chan@gmail.com
Subject: How will Hadoop handle it when a datanode server with total hardware failure?
Hi,
If each of my datanode servers has 8 hard disks (a 10-node cluster) and I use the default replication factor of 3, how will Hadoop handle it when a datanode with total hardware failure suddenly?
Regards
Arthur
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by Josh Baer <jb...@gmail.com>.
Three spindles failing on three different machines could potentially cause
data loss issues if they fail simultaneously-- which would be pretty
uncommon and affect only a small percentage of data. Otherwise if one
machine or individual spindle fails, the blocks that are stored on those
machines will get replicated on your remaining healthy machines after the
datanode is marked as dead by the namenode (this time is configurable in
settings). If the machine comes back online, you'll have 4 replications
temporarily.
You can read more about it here:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Data_Replication
On Sun, Apr 5, 2015 at 7:54 AM, daemeon reiydelle <da...@gmail.com>
wrote:
> With rep of 3 you would have to lose 3 entire nodes to lose data. The rep
> factor is 3 nodes, not 3 spindles.. The number of disks (sort of) determine
> how hdfs spreads io across the spindles for the single copy of the data
> (one of 3 nodes with copies) that the node owns. Note that things get
> slightly complicated when the FIRST datum is written to a cluster. (But
> that was not your question ; {)
> On Apr 4, 2015 10:39 PM, "Arthur Chan" <ar...@gmail.com> wrote:
>
>> Hi,
>>
>> I use the default replication factor 3 here, the cluster has 10 nodes,
>> each of my datanode has 8 hard disks. If one of the nodes is down because
>> of hardware failure, i.e. the 8 hard disks will no longer be available
>> immediately during the down time of this machine, does it mean that I will
>> have data lost? (8 hard disks > 3 replicated)
>>
>> Or what would be the maximum number of servers that are allowed to be
>> down without data lost here?
>>
>> Regards
>> Arthur
>>
>> On Wednesday, December 17, 2014, Harshit Mathur <ma...@gmail.com>
>> wrote:
>>
>>> Hi Arthur,
>>>
>>> In HDFS there will be block level replication, In case of total failure
>>> of a datanode the lost blocks will get under replicated hence the namenode
>>> will create copy of these under replicated blocks on some other datanode.
>>>
>>> BR,
>>> Harshit
>>>
>>> On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com <
>>> arthur.hk.chan@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> If each of my datanode servers has 8 hard disks (a 10-node cluster)
>>>> and I use the default replication factor of 3, how will Hadoop handle it
>>>> when a datanode with total hardware failure suddenly?
>>>>
>>>> Regards
>>>> Arthur
>>>>
>>>
>>>
>>>
>>> --
>>> Harshit Mathur
>>>
>>
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by Josh Baer <jb...@gmail.com>.
Three spindles failing on three different machines could potentially cause
data loss issues if they fail simultaneously-- which would be pretty
uncommon and affect only a small percentage of data. Otherwise if one
machine or individual spindle fails, the blocks that are stored on those
machines will get replicated on your remaining healthy machines after the
datanode is marked as dead by the namenode (this time is configurable in
settings). If the machine comes back online, you'll have 4 replications
temporarily.
You can read more about it here:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Data_Replication
On Sun, Apr 5, 2015 at 7:54 AM, daemeon reiydelle <da...@gmail.com>
wrote:
> With rep of 3 you would have to lose 3 entire nodes to lose data. The rep
> factor is 3 nodes, not 3 spindles.. The number of disks (sort of) determine
> how hdfs spreads io across the spindles for the single copy of the data
> (one of 3 nodes with copies) that the node owns. Note that things get
> slightly complicated when the FIRST datum is written to a cluster. (But
> that was not your question ; {)
> On Apr 4, 2015 10:39 PM, "Arthur Chan" <ar...@gmail.com> wrote:
>
>> Hi,
>>
>> I use the default replication factor 3 here, the cluster has 10 nodes,
>> each of my datanode has 8 hard disks. If one of the nodes is down because
>> of hardware failure, i.e. the 8 hard disks will no longer be available
>> immediately during the down time of this machine, does it mean that I will
>> have data lost? (8 hard disks > 3 replicated)
>>
>> Or what would be the maximum number of servers that are allowed to be
>> down without data lost here?
>>
>> Regards
>> Arthur
>>
>> On Wednesday, December 17, 2014, Harshit Mathur <ma...@gmail.com>
>> wrote:
>>
>>> Hi Arthur,
>>>
>>> In HDFS there will be block level replication, In case of total failure
>>> of a datanode the lost blocks will get under replicated hence the namenode
>>> will create copy of these under replicated blocks on some other datanode.
>>>
>>> BR,
>>> Harshit
>>>
>>> On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com <
>>> arthur.hk.chan@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> If each of my datanode servers has 8 hard disks (a 10-node cluster)
>>>> and I use the default replication factor of 3, how will Hadoop handle it
>>>> when a datanode with total hardware failure suddenly?
>>>>
>>>> Regards
>>>> Arthur
>>>>
>>>
>>>
>>>
>>> --
>>> Harshit Mathur
>>>
>>
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by Josh Baer <jb...@gmail.com>.
Three spindles failing on three different machines could potentially cause
data loss issues if they fail simultaneously-- which would be pretty
uncommon and affect only a small percentage of data. Otherwise if one
machine or individual spindle fails, the blocks that are stored on those
machines will get replicated on your remaining healthy machines after the
datanode is marked as dead by the namenode (this time is configurable in
settings). If the machine comes back online, you'll have 4 replications
temporarily.
You can read more about it here:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Data_Replication
On Sun, Apr 5, 2015 at 7:54 AM, daemeon reiydelle <da...@gmail.com>
wrote:
> With rep of 3 you would have to lose 3 entire nodes to lose data. The rep
> factor is 3 nodes, not 3 spindles.. The number of disks (sort of) determine
> how hdfs spreads io across the spindles for the single copy of the data
> (one of 3 nodes with copies) that the node owns. Note that things get
> slightly complicated when the FIRST datum is written to a cluster. (But
> that was not your question ; {)
> On Apr 4, 2015 10:39 PM, "Arthur Chan" <ar...@gmail.com> wrote:
>
>> Hi,
>>
>> I use the default replication factor 3 here, the cluster has 10 nodes,
>> each of my datanode has 8 hard disks. If one of the nodes is down because
>> of hardware failure, i.e. the 8 hard disks will no longer be available
>> immediately during the down time of this machine, does it mean that I will
>> have data lost? (8 hard disks > 3 replicated)
>>
>> Or what would be the maximum number of servers that are allowed to be
>> down without data lost here?
>>
>> Regards
>> Arthur
>>
>> On Wednesday, December 17, 2014, Harshit Mathur <ma...@gmail.com>
>> wrote:
>>
>>> Hi Arthur,
>>>
>>> In HDFS there will be block level replication, In case of total failure
>>> of a datanode the lost blocks will get under replicated hence the namenode
>>> will create copy of these under replicated blocks on some other datanode.
>>>
>>> BR,
>>> Harshit
>>>
>>> On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com <
>>> arthur.hk.chan@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> If each of my datanode servers has 8 hard disks (a 10-node cluster)
>>>> and I use the default replication factor of 3, how will Hadoop handle it
>>>> when a datanode with total hardware failure suddenly?
>>>>
>>>> Regards
>>>> Arthur
>>>>
>>>
>>>
>>>
>>> --
>>> Harshit Mathur
>>>
>>
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by Josh Baer <jb...@gmail.com>.
Three spindles failing on three different machines could potentially cause
data loss issues if they fail simultaneously-- which would be pretty
uncommon and affect only a small percentage of data. Otherwise if one
machine or individual spindle fails, the blocks that are stored on those
machines will get replicated on your remaining healthy machines after the
datanode is marked as dead by the namenode (this time is configurable in
settings). If the machine comes back online, you'll have 4 replications
temporarily.
You can read more about it here:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Data_Replication
On Sun, Apr 5, 2015 at 7:54 AM, daemeon reiydelle <da...@gmail.com>
wrote:
> With rep of 3 you would have to lose 3 entire nodes to lose data. The rep
> factor is 3 nodes, not 3 spindles.. The number of disks (sort of) determine
> how hdfs spreads io across the spindles for the single copy of the data
> (one of 3 nodes with copies) that the node owns. Note that things get
> slightly complicated when the FIRST datum is written to a cluster. (But
> that was not your question ; {)
> On Apr 4, 2015 10:39 PM, "Arthur Chan" <ar...@gmail.com> wrote:
>
>> Hi,
>>
>> I use the default replication factor 3 here, the cluster has 10 nodes,
>> each of my datanode has 8 hard disks. If one of the nodes is down because
>> of hardware failure, i.e. the 8 hard disks will no longer be available
>> immediately during the down time of this machine, does it mean that I will
>> have data lost? (8 hard disks > 3 replicated)
>>
>> Or what would be the maximum number of servers that are allowed to be
>> down without data lost here?
>>
>> Regards
>> Arthur
>>
>> On Wednesday, December 17, 2014, Harshit Mathur <ma...@gmail.com>
>> wrote:
>>
>>> Hi Arthur,
>>>
>>> In HDFS there will be block level replication, In case of total failure
>>> of a datanode the lost blocks will get under replicated hence the namenode
>>> will create copy of these under replicated blocks on some other datanode.
>>>
>>> BR,
>>> Harshit
>>>
>>> On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com <
>>> arthur.hk.chan@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> If each of my datanode servers has 8 hard disks (a 10-node cluster)
>>>> and I use the default replication factor of 3, how will Hadoop handle it
>>>> when a datanode with total hardware failure suddenly?
>>>>
>>>> Regards
>>>> Arthur
>>>>
>>>
>>>
>>>
>>> --
>>> Harshit Mathur
>>>
>>
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by daemeon reiydelle <da...@gmail.com>.
With rep of 3 you would have to lose 3 entire nodes to lose data. The rep
factor is 3 nodes, not 3 spindles.. The number of disks (sort of) determine
how hdfs spreads io across the spindles for the single copy of the data
(one of 3 nodes with copies) that the node owns. Note that things get
slightly complicated when the FIRST datum is written to a cluster. (But
that was not your question ; {)
On Apr 4, 2015 10:39 PM, "Arthur Chan" <ar...@gmail.com> wrote:
> Hi,
>
> I use the default replication factor 3 here, the cluster has 10 nodes,
> each of my datanode has 8 hard disks. If one of the nodes is down because
> of hardware failure, i.e. the 8 hard disks will no longer be available
> immediately during the down time of this machine, does it mean that I will
> have data lost? (8 hard disks > 3 replicated)
>
> Or what would be the maximum number of servers that are allowed to be down
> without data lost here?
>
> Regards
> Arthur
>
> On Wednesday, December 17, 2014, Harshit Mathur <ma...@gmail.com>
> wrote:
>
>> Hi Arthur,
>>
>> In HDFS there will be block level replication, In case of total failure
>> of a datanode the lost blocks will get under replicated hence the namenode
>> will create copy of these under replicated blocks on some other datanode.
>>
>> BR,
>> Harshit
>>
>> On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com <
>> arthur.hk.chan@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> If each of my datanode servers has 8 hard disks (a 10-node cluster) and
>>> I use the default replication factor of 3, how will Hadoop handle it when a
>>> datanode with total hardware failure suddenly?
>>>
>>> Regards
>>> Arthur
>>>
>>
>>
>>
>> --
>> Harshit Mathur
>>
>
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by daemeon reiydelle <da...@gmail.com>.
With rep of 3 you would have to lose 3 entire nodes to lose data. The rep
factor is 3 nodes, not 3 spindles.. The number of disks (sort of) determine
how hdfs spreads io across the spindles for the single copy of the data
(one of 3 nodes with copies) that the node owns. Note that things get
slightly complicated when the FIRST datum is written to a cluster. (But
that was not your question ; {)
On Apr 4, 2015 10:39 PM, "Arthur Chan" <ar...@gmail.com> wrote:
> Hi,
>
> I use the default replication factor 3 here, the cluster has 10 nodes,
> each of my datanode has 8 hard disks. If one of the nodes is down because
> of hardware failure, i.e. the 8 hard disks will no longer be available
> immediately during the down time of this machine, does it mean that I will
> have data lost? (8 hard disks > 3 replicated)
>
> Or what would be the maximum number of servers that are allowed to be down
> without data lost here?
>
> Regards
> Arthur
>
> On Wednesday, December 17, 2014, Harshit Mathur <ma...@gmail.com>
> wrote:
>
>> Hi Arthur,
>>
>> In HDFS there will be block level replication, In case of total failure
>> of a datanode the lost blocks will get under replicated hence the namenode
>> will create copy of these under replicated blocks on some other datanode.
>>
>> BR,
>> Harshit
>>
>> On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com <
>> arthur.hk.chan@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> If each of my datanode servers has 8 hard disks (a 10-node cluster) and
>>> I use the default replication factor of 3, how will Hadoop handle it when a
>>> datanode with total hardware failure suddenly?
>>>
>>> Regards
>>> Arthur
>>>
>>
>>
>>
>> --
>> Harshit Mathur
>>
>
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by daemeon reiydelle <da...@gmail.com>.
With rep of 3 you would have to lose 3 entire nodes to lose data. The rep
factor is 3 nodes, not 3 spindles.. The number of disks (sort of) determine
how hdfs spreads io across the spindles for the single copy of the data
(one of 3 nodes with copies) that the node owns. Note that things get
slightly complicated when the FIRST datum is written to a cluster. (But
that was not your question ; {)
On Apr 4, 2015 10:39 PM, "Arthur Chan" <ar...@gmail.com> wrote:
> Hi,
>
> I use the default replication factor 3 here, the cluster has 10 nodes,
> each of my datanode has 8 hard disks. If one of the nodes is down because
> of hardware failure, i.e. the 8 hard disks will no longer be available
> immediately during the down time of this machine, does it mean that I will
> have data lost? (8 hard disks > 3 replicated)
>
> Or what would be the maximum number of servers that are allowed to be down
> without data lost here?
>
> Regards
> Arthur
>
> On Wednesday, December 17, 2014, Harshit Mathur <ma...@gmail.com>
> wrote:
>
>> Hi Arthur,
>>
>> In HDFS there will be block level replication, In case of total failure
>> of a datanode the lost blocks will get under replicated hence the namenode
>> will create copy of these under replicated blocks on some other datanode.
>>
>> BR,
>> Harshit
>>
>> On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com <
>> arthur.hk.chan@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> If each of my datanode servers has 8 hard disks (a 10-node cluster) and
>>> I use the default replication factor of 3, how will Hadoop handle it when a
>>> datanode with total hardware failure suddenly?
>>>
>>> Regards
>>> Arthur
>>>
>>
>>
>>
>> --
>> Harshit Mathur
>>
>
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by daemeon reiydelle <da...@gmail.com>.
With rep of 3 you would have to lose 3 entire nodes to lose data. The rep
factor is 3 nodes, not 3 spindles.. The number of disks (sort of) determine
how hdfs spreads io across the spindles for the single copy of the data
(one of 3 nodes with copies) that the node owns. Note that things get
slightly complicated when the FIRST datum is written to a cluster. (But
that was not your question ; {)
On Apr 4, 2015 10:39 PM, "Arthur Chan" <ar...@gmail.com> wrote:
> Hi,
>
> I use the default replication factor 3 here, the cluster has 10 nodes,
> each of my datanode has 8 hard disks. If one of the nodes is down because
> of hardware failure, i.e. the 8 hard disks will no longer be available
> immediately during the down time of this machine, does it mean that I will
> have data lost? (8 hard disks > 3 replicated)
>
> Or what would be the maximum number of servers that are allowed to be down
> without data lost here?
>
> Regards
> Arthur
>
> On Wednesday, December 17, 2014, Harshit Mathur <ma...@gmail.com>
> wrote:
>
>> Hi Arthur,
>>
>> In HDFS there will be block level replication, In case of total failure
>> of a datanode the lost blocks will get under replicated hence the namenode
>> will create copy of these under replicated blocks on some other datanode.
>>
>> BR,
>> Harshit
>>
>> On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com <
>> arthur.hk.chan@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> If each of my datanode servers has 8 hard disks (a 10-node cluster) and
>>> I use the default replication factor of 3, how will Hadoop handle it when a
>>> datanode with total hardware failure suddenly?
>>>
>>> Regards
>>> Arthur
>>>
>>
>>
>>
>> --
>> Harshit Mathur
>>
>
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by Arthur Chan <ar...@gmail.com>.
Hi,
I use the default replication factor 3 here, the cluster has 10 nodes, each
of my datanode has 8 hard disks. If one of the nodes is down because of
hardware failure, i.e. the 8 hard disks will no longer be available
immediately during the down time of this machine, does it mean that I will
have data lost? (8 hard disks > 3 replicated)
Or what would be the maximum number of servers that are allowed to be down
without data lost here?
Regards
Arthur
On Wednesday, December 17, 2014, Harshit Mathur <ma...@gmail.com>
wrote:
> Hi Arthur,
>
> In HDFS there will be block level replication, In case of total failure of
> a datanode the lost blocks will get under replicated hence the namenode
> will create copy of these under replicated blocks on some other datanode.
>
> BR,
> Harshit
>
> On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com
> <javascript:_e(%7B%7D,'cvml','Arthur.hk.chan@gmail.com');> <
> arthur.hk.chan@gmail.com
> <javascript:_e(%7B%7D,'cvml','arthur.hk.chan@gmail.com');>> wrote:
>>
>> Hi,
>>
>> If each of my datanode servers has 8 hard disks (a 10-node cluster) and
>> I use the default replication factor of 3, how will Hadoop handle it when a
>> datanode with total hardware failure suddenly?
>>
>> Regards
>> Arthur
>>
>
>
>
> --
> Harshit Mathur
>
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by Arthur Chan <ar...@gmail.com>.
Hi,
I use the default replication factor 3 here, the cluster has 10 nodes, each
of my datanode has 8 hard disks. If one of the nodes is down because of
hardware failure, i.e. the 8 hard disks will no longer be available
immediately during the down time of this machine, does it mean that I will
have data lost? (8 hard disks > 3 replicated)
Or what would be the maximum number of servers that are allowed to be down
without data lost here?
Regards
Arthur
On Wednesday, December 17, 2014, Harshit Mathur <ma...@gmail.com>
wrote:
> Hi Arthur,
>
> In HDFS there will be block level replication, In case of total failure of
> a datanode the lost blocks will get under replicated hence the namenode
> will create copy of these under replicated blocks on some other datanode.
>
> BR,
> Harshit
>
> On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com
> <javascript:_e(%7B%7D,'cvml','Arthur.hk.chan@gmail.com');> <
> arthur.hk.chan@gmail.com
> <javascript:_e(%7B%7D,'cvml','arthur.hk.chan@gmail.com');>> wrote:
>>
>> Hi,
>>
>> If each of my datanode servers has 8 hard disks (a 10-node cluster) and
>> I use the default replication factor of 3, how will Hadoop handle it when a
>> datanode with total hardware failure suddenly?
>>
>> Regards
>> Arthur
>>
>
>
>
> --
> Harshit Mathur
>
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by Arthur Chan <ar...@gmail.com>.
Hi,
I use the default replication factor 3 here, the cluster has 10 nodes, each
of my datanode has 8 hard disks. If one of the nodes is down because of
hardware failure, i.e. the 8 hard disks will no longer be available
immediately during the down time of this machine, does it mean that I will
have data lost? (8 hard disks > 3 replicated)
Or what would be the maximum number of servers that are allowed to be down
without data lost here?
Regards
Arthur
On Wednesday, December 17, 2014, Harshit Mathur <ma...@gmail.com>
wrote:
> Hi Arthur,
>
> In HDFS there will be block level replication, In case of total failure of
> a datanode the lost blocks will get under replicated hence the namenode
> will create copy of these under replicated blocks on some other datanode.
>
> BR,
> Harshit
>
> On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com
> <javascript:_e(%7B%7D,'cvml','Arthur.hk.chan@gmail.com');> <
> arthur.hk.chan@gmail.com
> <javascript:_e(%7B%7D,'cvml','arthur.hk.chan@gmail.com');>> wrote:
>>
>> Hi,
>>
>> If each of my datanode servers has 8 hard disks (a 10-node cluster) and
>> I use the default replication factor of 3, how will Hadoop handle it when a
>> datanode with total hardware failure suddenly?
>>
>> Regards
>> Arthur
>>
>
>
>
> --
> Harshit Mathur
>
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by Arthur Chan <ar...@gmail.com>.
Hi,
I use the default replication factor 3 here, the cluster has 10 nodes, each
of my datanode has 8 hard disks. If one of the nodes is down because of
hardware failure, i.e. the 8 hard disks will no longer be available
immediately during the down time of this machine, does it mean that I will
have data lost? (8 hard disks > 3 replicated)
Or what would be the maximum number of servers that are allowed to be down
without data lost here?
Regards
Arthur
On Wednesday, December 17, 2014, Harshit Mathur <ma...@gmail.com>
wrote:
> Hi Arthur,
>
> In HDFS there will be block level replication, In case of total failure of
> a datanode the lost blocks will get under replicated hence the namenode
> will create copy of these under replicated blocks on some other datanode.
>
> BR,
> Harshit
>
> On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com
> <javascript:_e(%7B%7D,'cvml','Arthur.hk.chan@gmail.com');> <
> arthur.hk.chan@gmail.com
> <javascript:_e(%7B%7D,'cvml','arthur.hk.chan@gmail.com');>> wrote:
>>
>> Hi,
>>
>> If each of my datanode servers has 8 hard disks (a 10-node cluster) and
>> I use the default replication factor of 3, how will Hadoop handle it when a
>> datanode with total hardware failure suddenly?
>>
>> Regards
>> Arthur
>>
>
>
>
> --
> Harshit Mathur
>
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by Harshit Mathur <ma...@gmail.com>.
Hi Arthur,
In HDFS there will be block level replication, In case of total failure of
a datanode the lost blocks will get under replicated hence the namenode
will create copy of these under replicated blocks on some other datanode.
BR,
Harshit
On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com <
arthur.hk.chan@gmail.com> wrote:
>
> Hi,
>
> If each of my datanode servers has 8 hard disks (a 10-node cluster) and I
> use the default replication factor of 3, how will Hadoop handle it when a
> datanode with total hardware failure suddenly?
>
> Regards
> Arthur
>
--
Harshit Mathur
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by Harshit Mathur <ma...@gmail.com>.
Hi Arthur,
In HDFS there will be block level replication, In case of total failure of
a datanode the lost blocks will get under replicated hence the namenode
will create copy of these under replicated blocks on some other datanode.
BR,
Harshit
On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com <
arthur.hk.chan@gmail.com> wrote:
>
> Hi,
>
> If each of my datanode servers has 8 hard disks (a 10-node cluster) and I
> use the default replication factor of 3, how will Hadoop handle it when a
> datanode with total hardware failure suddenly?
>
> Regards
> Arthur
>
--
Harshit Mathur
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by Harshit Mathur <ma...@gmail.com>.
Hi Arthur,
In HDFS there will be block level replication, In case of total failure of
a datanode the lost blocks will get under replicated hence the namenode
will create copy of these under replicated blocks on some other datanode.
BR,
Harshit
On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com <
arthur.hk.chan@gmail.com> wrote:
>
> Hi,
>
> If each of my datanode servers has 8 hard disks (a 10-node cluster) and I
> use the default replication factor of 3, how will Hadoop handle it when a
> datanode with total hardware failure suddenly?
>
> Regards
> Arthur
>
--
Harshit Mathur
Re: How will Hadoop handle it when a datanode server with total hardware failure?
Posted by zhangyujian1984 <zh...@gmail.com>.
i think hadoop treats a node as a minimal unit, which means a internal failure will be regards as a node failure.
datanode uses heartbeat to keep block alive, if any block failure was detected by namenode, it will be recovered
in another datanode.
Yujian Zhang
Southeast University, Nanjing, China
From: Arthur.hk.chan@gmail.com
Date: 2014-12-17 14:05
To: <us...@hadoop.apache.org>
CC: Arthur.hk.chan@gmail.com
Subject: How will Hadoop handle it when a datanode server with total hardware failure?
Hi,
If each of my datanode servers has 8 hard disks (a 10-node cluster) and I use the default replication factor of 3, how will Hadoop handle it when a datanode with total hardware failure suddenly?
Regards
Arthur
Re: How will Hadoop handle it when a datanode server with total hardware failure?
Posted by zhangyujian1984 <zh...@gmail.com>.
i think hadoop treats a node as a minimal unit, which means a internal failure will be regards as a node failure.
datanode uses heartbeat to keep block alive, if any block failure was detected by namenode, it will be recovered
in another datanode.
Yujian Zhang
Southeast University, Nanjing, China
From: Arthur.hk.chan@gmail.com
Date: 2014-12-17 14:05
To: <us...@hadoop.apache.org>
CC: Arthur.hk.chan@gmail.com
Subject: How will Hadoop handle it when a datanode server with total hardware failure?
Hi,
If each of my datanode servers has 8 hard disks (a 10-node cluster) and I use the default replication factor of 3, how will Hadoop handle it when a datanode with total hardware failure suddenly?
Regards
Arthur
Re: How will Hadoop handle it when a datanode server with total
hardware failure?
Posted by Harshit Mathur <ma...@gmail.com>.
Hi Arthur,
In HDFS there will be block level replication, In case of total failure of
a datanode the lost blocks will get under replicated hence the namenode
will create copy of these under replicated blocks on some other datanode.
BR,
Harshit
On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com <
arthur.hk.chan@gmail.com> wrote:
>
> Hi,
>
> If each of my datanode servers has 8 hard disks (a 10-node cluster) and I
> use the default replication factor of 3, how will Hadoop handle it when a
> datanode with total hardware failure suddenly?
>
> Regards
> Arthur
>
--
Harshit Mathur