You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mayuran Yogarajah <ma...@casalemedia.com> on 2009/08/11 20:09:07 UTC
NN + secondary got full, even though data nodes had plenty of space
I have a 6 node cluster running Hadoop 0.18.3. I'm trying to figure out
how the data was spread out like this:
node001 94.15%
node002 94.16%
node003 48.22%
node004 47.85%
node005 48.12%
node006 43.18%
Node 001 (NN) and node 002( secondary NN) both got full, while the other
data nodes had more space left. I had assumed that Hadoop would distribute
more blocks to nodes 3-6 since they had much more space, but it ended up
filling up nodes1 and 2. Is this expected?
thanks,
M
Re: NN + secondary got full, even though data nodes had plenty of
space
Posted by Mayuran Yogarajah <ma...@casalemedia.com>.
Todd Lipcon wrote:
> Hi Mayuran,
>
> Do you do all of your uploads of data into your Hadoop cluster from node001
> and node002?
>
> If so, keep in mind that one of your replicas will always be written on
> localhost in the case that it is part of the cluster.
>
> You should consider running the rebalancer to even up your space usage.
>
> -Todd
>
>
Actually yes I have been doing this. I'll try rebalancer, thanks for
your help.
M
> On Tue, Aug 11, 2009 at 11:09 AM, Mayuran Yogarajah <
> mayuran.yogarajah@casalemedia.com> wrote:
>
>
>> I have a 6 node cluster running Hadoop 0.18.3. I'm trying to figure out
>> how the data was spread out like this:
>>
>> node001 94.15%
>> node002 94.16%
>> node003 48.22%
>> node004 47.85%
>> node005 48.12%
>> node006 43.18%
>> Node 001 (NN) and node 002( secondary NN) both got full, while the other
>> data nodes had more space left. I had assumed that Hadoop would distribute
>> more blocks to nodes 3-6 since they had much more space, but it ended up
>> filling up nodes1 and 2. Is this expected?
>>
>> thanks,
>> M
>>
>>
>>
Re: NN + secondary got full, even though data nodes had plenty of
space
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Mayuran,
Do you do all of your uploads of data into your Hadoop cluster from node001
and node002?
If so, keep in mind that one of your replicas will always be written on
localhost in the case that it is part of the cluster.
You should consider running the rebalancer to even up your space usage.
-Todd
On Tue, Aug 11, 2009 at 11:09 AM, Mayuran Yogarajah <
mayuran.yogarajah@casalemedia.com> wrote:
> I have a 6 node cluster running Hadoop 0.18.3. I'm trying to figure out
> how the data was spread out like this:
>
> node001 94.15%
> node002 94.16%
> node003 48.22%
> node004 47.85%
> node005 48.12%
> node006 43.18%
> Node 001 (NN) and node 002( secondary NN) both got full, while the other
> data nodes had more space left. I had assumed that Hadoop would distribute
> more blocks to nodes 3-6 since they had much more space, but it ended up
> filling up nodes1 and 2. Is this expected?
>
> thanks,
> M
>
>
Re: NN + secondary got full, even though data nodes had plenty of
space
Posted by Mayuran Yogarajah <ma...@casalemedia.com>.
Amandeep Khurana wrote:
> Is your NN doubling up as a DN? If its not, I wonder how the NN is full...
>
Yes both NN and secondary are doubling as DN, sorry I should have mentioned
this earlier.
M
> On 8/11/09, Mayuran Yogarajah <ma...@casalemedia.com> wrote:
>
>> I have a 6 node cluster running Hadoop 0.18.3. I'm trying to figure out
>> how the data was spread out like this:
>>
>> node001 94.15%
>> node002 94.16%
>> node003 48.22%
>> node004 47.85%
>> node005 48.12%
>> node006 43.18%
>>
>> Node 001 (NN) and node 002( secondary NN) both got full, while the other
>> data nodes had more space left. I had assumed that Hadoop would distribute
>> more blocks to nodes 3-6 since they had much more space, but it ended up
>> filling up nodes1 and 2. Is this expected?
>>
>> thanks,
>> M
>>
>>
>>
>
>
> --
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
Re: NN + secondary got full, even though data nodes had plenty of
space
Posted by Amandeep Khurana <am...@gmail.com>.
Is your NN doubling up as a DN? If its not, I wonder how the NN is full...
On 8/11/09, Mayuran Yogarajah <ma...@casalemedia.com> wrote:
> I have a 6 node cluster running Hadoop 0.18.3. I'm trying to figure out
> how the data was spread out like this:
>
> node001 94.15%
> node002 94.16%
> node003 48.22%
> node004 47.85%
> node005 48.12%
> node006 43.18%
>
> Node 001 (NN) and node 002( secondary NN) both got full, while the other
> data nodes had more space left. I had assumed that Hadoop would distribute
> more blocks to nodes 3-6 since they had much more space, but it ended up
> filling up nodes1 and 2. Is this expected?
>
> thanks,
> M
>
>
--
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz