You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Raj V <ra...@yahoo.com> on 2010/08/05 17:33:26 UTC
hdfs space problem.
I run a 512 node hadoop cluster. Yesterday I moved 30Gb of compressed data from
a NFS mounted partition by running on the namenode
hadoop fs -copyFromLocal /mnt/data/data1 /mnt/data/data2 mnt/data/data3
hdfs:/data
When the job completed the local disk on the namenode was 40% full ( Most of it
used by the dfs dierctories) while the others had 1% disk utilization.
Just to see if there was an issue, I deleted the hdfs:/data directory and
restarted the move from a datanode.
Once again the disk space on that data node was substantially over utilized.
I would have assumed that the disk space would be more or less uniformly
consumed on all the data nodes.
Is there a reason why one disk would be over utilized?
Do I have to run balancer everytime I copy data?
Am I missing something?
Raj
centralized record reader in new API
Posted by Gang Luo <lg...@yahoo.com.cn>.
Hi all,
to create a RecordReader in new API, we needs a TaskAttemptContext object, which
seems to me the RecordReader should only be created on each split that has been
assigned a task ID. However, I want to do a centralized sampling and create
record reader on some splits before the job is submitted. What I am doing is
create a dummy TaskAttemptContext and use it to create record reader, but not
sure whether there is some side-effects. Is there any better way to do this? Why
we are not supposed to create record reader centrally as indicated by the new
API?
Thanks,
-Gang
Re: hdfs space problem.
Posted by Steve Loughran <st...@apache.org>.
On 05/08/10 19:28, Raj V wrote:
> Thank you. I realized that I was running the datanode on the namenode and
> stopped it, but did not know that the first copy went to the local node.
>
> Raj
It's a placement decision that makes sense for code running as MR jobs,
ensuring that the output of work goes to the local machine and not
somewhere random, but on big imports like your's you get penalised.
Some datacentres have one or two IO nodes in the cluster that aren't
running hadoop HDFS or task trackers, but let you get at the data at
full datacentre rates, just to help with these kind of problems.
Otherwies, if you can implement your import as a MapReduce job, Hadoop
can do the work for you
-steve
Re: hdfs space problem.
Posted by Raj V <ra...@yahoo.com>.
Thank you. I realized that I was running the datanode on the namenode and
stopped it, but did not know that the first copy went to the local node.
Raj
________________________________
From: Dmitry Pushkarev <um...@stanford.edu>
To: common-user@hadoop.apache.org
Sent: Thu, August 5, 2010 11:02:08 AM
Subject: RE: hdfs space problem.
when you copy files and have a local datanode - first copy will end up
there.
Just stop datanode at the node from which you copy files, and they will end
up on random nodes.
Also don't run datanode at the same machine as namenode.
-----Original Message-----
From: Raj V [mailto:rajvish@yahoo.com]
Sent: Thursday, August 05, 2010 8:33 AM
To: common-user@hadoop.apache.org
Subject: hdfs space problem.
I run a 512 node hadoop cluster. Yesterday I moved 30Gb of compressed data
from
a NFS mounted partition by running on the namenode
hadoop fs -copyFromLocal /mnt/data/data1 /mnt/data/data2 mnt/data/data3
hdfs:/data
When the job completed the local disk on the namenode was 40% full ( Most of
it
used by the dfs dierctories) while the others had 1% disk utilization.
Just to see if there was an issue, I deleted the hdfs:/data directory and
restarted the move from a datanode.
Once again the disk space on that data node was substantially over utilized.
I would have assumed that the disk space would be more or less uniformly
consumed on all the data nodes.
Is there a reason why one disk would be over utilized?
Do I have to run balancer everytime I copy data?
Am I missing something?
Raj
RE: hdfs space problem.
Posted by Dmitry Pushkarev <um...@stanford.edu>.
when you copy files and have a local datanode - first copy will end up
there.
Just stop datanode at the node from which you copy files, and they will end
up on random nodes.
Also don't run datanode at the same machine as namenode.
-----Original Message-----
From: Raj V [mailto:rajvish@yahoo.com]
Sent: Thursday, August 05, 2010 8:33 AM
To: common-user@hadoop.apache.org
Subject: hdfs space problem.
I run a 512 node hadoop cluster. Yesterday I moved 30Gb of compressed data
from
a NFS mounted partition by running on the namenode
hadoop fs -copyFromLocal /mnt/data/data1 /mnt/data/data2 mnt/data/data3
hdfs:/data
When the job completed the local disk on the namenode was 40% full ( Most of
it
used by the dfs dierctories) while the others had 1% disk utilization.
Just to see if there was an issue, I deleted the hdfs:/data directory and
restarted the move from a datanode.
Once again the disk space on that data node was substantially over utilized.
I would have assumed that the disk space would be more or less uniformly
consumed on all the data nodes.
Is there a reason why one disk would be over utilized?
Do I have to run balancer everytime I copy data?
Am I missing something?
Raj