You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by ilayaraja <il...@rediff.co.in> on 2009/03/30 07:24:57 UTC
Problem: Some blocks remain under replicated
Hello !
I am trying to increase the replication factor of a directory in our hadoop dfs from 1 to 2.
I observe that some of the blocks (12 out of 400) always remain under replicated, throwing the following message when I do an 'fsck' :
Under replicated blk_9084408236031628003. Target Replicas is 2 but found 1 replica(s).
I thought it could be a problem with a specific data node in the cluster, however I observe that the under replicated blocks belong to different data nodes .
Please give me your thoughts.
Thanks.
Ilay
Re: Problem: Some blocks remain under replicated
Posted by ilayaraja <il...@rediff.co.in>.
We are using hadoop-0.15.
Let me explain the scenario:
We have around 6 TB of data in our cluster on couple of data
directories(/mnt, /mnt2)with a replication factor of 1. when we increased
the replication to 2 for the entire data, we observed that /mnt is used 100%
while /mnt2 is under utilized. So we wanted to balance the utilization of
space from both the data directories by changing the hadoop code for
getNextVolume(..) API. The new algorithm checks which data directory is
having more space available and returns that as the volume for the block to
be written. This updated version of hadoop is then used for setting the
replication to 2 for the entire dfs. However, when the dfs replication is
over, it reported more than 100 GB of data blocks are missing as well as
some blocks are under replicated. We also observed that there are many
blocks of size zero present in the cluster, we do not know how these blocks
were created.
----- Original Message -----
From: "Hairong Kuang" <ha...@yahoo-inc.com>
To: "hadoop-dev" <co...@hadoop.apache.org>; "ilayaraja"
<il...@rediff.co.in>; "hadoop-user" <co...@hadoop.apache.org>
Sent: Tuesday, March 31, 2009 3:30 AM
Subject: Re: Problem: Some blocks remain under replicated
> Which version of HADOOP are you running? Your cluster might have hit
> HADOOP-5465.
>
> Hairong
>
>
> On 3/29/09 10:24 PM, "ilayaraja" <il...@rediff.co.in> wrote:
>
>> Hello !
>>
>> I am trying to increase the replication factor of a directory in our
>> hadoop
>> dfs from 1 to 2.
>> I observe that some of the blocks (12 out of 400) always remain under
>> replicated, throwing the following message when I do an 'fsck' :
>>
>> Under replicated blk_9084408236031628003. Target
>> Replicas
>> is 2 but found 1 replica(s).
>>
>> I thought it could be a problem with a specific data node in the cluster,
>> however I observe that the under replicated blocks belong to different
>> data
>> nodes .
>>
>> Please give me your thoughts.
>>
>> Thanks.
>> Ilay
>>
>>
>>
>>
>
Re: Problem: Some blocks remain under replicated
Posted by ilayaraja <il...@rediff.co.in>.
We are using hadoop-0.15.
Let me explain the scenario:
We have around 6 TB of data in our cluster on couple of data
directories(/mnt, /mnt2)with a replication factor of 1. when we increased
the replication to 2 for the entire data, we observed that /mnt is used 100%
while /mnt2 is under utilized. So we wanted to balance the utilization of
space from both the data directories by changing the hadoop code for
getNextVolume(..) API. The new algorithm checks which data directory is
having more space available and returns that as the volume for the block to
be written. This updated version of hadoop is then used for setting the
replication to 2 for the entire dfs. However, when the dfs replication is
over, it reported more than 100 GB of data blocks are missing as well as
some blocks are under replicated. We also observed that there are many
blocks of size zero present in the cluster, we do not know how these blocks
were created.
----- Original Message -----
From: "Hairong Kuang" <ha...@yahoo-inc.com>
To: "hadoop-dev" <co...@hadoop.apache.org>; "ilayaraja"
<il...@rediff.co.in>; "hadoop-user" <co...@hadoop.apache.org>
Sent: Tuesday, March 31, 2009 3:30 AM
Subject: Re: Problem: Some blocks remain under replicated
> Which version of HADOOP are you running? Your cluster might have hit
> HADOOP-5465.
>
> Hairong
>
>
> On 3/29/09 10:24 PM, "ilayaraja" <il...@rediff.co.in> wrote:
>
>> Hello !
>>
>> I am trying to increase the replication factor of a directory in our
>> hadoop
>> dfs from 1 to 2.
>> I observe that some of the blocks (12 out of 400) always remain under
>> replicated, throwing the following message when I do an 'fsck' :
>>
>> Under replicated blk_9084408236031628003. Target
>> Replicas
>> is 2 but found 1 replica(s).
>>
>> I thought it could be a problem with a specific data node in the cluster,
>> however I observe that the under replicated blocks belong to different
>> data
>> nodes .
>>
>> Please give me your thoughts.
>>
>> Thanks.
>> Ilay
>>
>>
>>
>>
>
Re: Problem: Some blocks remain under replicated
Posted by Hairong Kuang <ha...@yahoo-inc.com>.
Which version of HADOOP are you running? Your cluster might have hit
HADOOP-5465.
Hairong
On 3/29/09 10:24 PM, "ilayaraja" <il...@rediff.co.in> wrote:
> Hello !
>
> I am trying to increase the replication factor of a directory in our hadoop
> dfs from 1 to 2.
> I observe that some of the blocks (12 out of 400) always remain under
> replicated, throwing the following message when I do an 'fsck' :
>
> Under replicated blk_9084408236031628003. Target Replicas
> is 2 but found 1 replica(s).
>
> I thought it could be a problem with a specific data node in the cluster,
> however I observe that the under replicated blocks belong to different data
> nodes .
>
> Please give me your thoughts.
>
> Thanks.
> Ilay
>
>
>
>
Re: Problem: Some blocks remain under replicated
Posted by Hairong Kuang <ha...@yahoo-inc.com>.
Which version of HADOOP are you running? Your cluster might have hit
HADOOP-5465.
Hairong
On 3/29/09 10:24 PM, "ilayaraja" <il...@rediff.co.in> wrote:
> Hello !
>
> I am trying to increase the replication factor of a directory in our hadoop
> dfs from 1 to 2.
> I observe that some of the blocks (12 out of 400) always remain under
> replicated, throwing the following message when I do an 'fsck' :
>
> Under replicated blk_9084408236031628003. Target Replicas
> is 2 but found 1 replica(s).
>
> I thought it could be a problem with a specific data node in the cluster,
> however I observe that the under replicated blocks belong to different data
> nodes .
>
> Please give me your thoughts.
>
> Thanks.
> Ilay
>
>
>
>