You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Alexander Frolov <al...@gmail.com> on 2014/02/07 09:41:07 UTC
Fwd: HDFS: file is not distributed after upload
Hi, folks!
I've deployed hadoop (0.20.203.0rc1) on 8-node cluster. After uploading
file onto hdfs I've got this file only on one of the nodes instead of being
uniformly distributed across all nodes. What can be the issue?
$HADOOP_HOME/bin/hadoop dfs -copyFromLocal ../data/rmat-20.0
/user/frolo/input/rmat-20.0
$HADOOP_HOME/bin/hadoop dfs -stat "%b %o %r %n" /user/frolo/input/rmat-*
1220222968 67108864 1 rmat-20.0
$HADOOP_HOME/bin/hadoop dfsadmin -report
Configured Capacity: 2536563998720 (2.31 TB)
Present Capacity: 1642543419392 (1.49 TB)
DFS Remaining: 1641312030720 (1.49 TB)
DFS Used: 1231388672 (1.15 GB)
DFS Used%: 0.07%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.10.1.15:50010
Decommission Status : Normal
Configured Capacity: 317070499840 (295.29 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 131536928768 (122.5 GB)
DFS Remaining: 185533546496(172.79 GB)
DFS Used%: 0%
DFS Remaining%: 58.51%
Last contact: Fri Feb 07 12:10:27 MSK 2014
Name: 10.10.1.13:50010
Decommission Status : Normal
Configured Capacity: 317070499840 (295.29 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 131533377536 (122.5 GB)
DFS Remaining: 185537097728(172.79 GB)
DFS Used%: 0%
DFS Remaining%: 58.52%
Last contact: Fri Feb 07 12:10:27 MSK 2014
Name: 10.10.1.17:50010
Decommission Status : Normal
Configured Capacity: 317070499840 (295.29 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 120023924736 (111.78 GB)
DFS Remaining: 197046550528(183.51 GB)
DFS Used%: 0%
DFS Remaining%: 62.15%
Last contact: Fri Feb 07 12:10:27 MSK 2014
Name: 10.10.1.18:50010
Decommission Status : Normal
Configured Capacity: 317070499840 (295.29 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 78510628864 (73.12 GB)
DFS Remaining: 238559846400(222.18 GB)
DFS Used%: 0%
DFS Remaining%: 75.24%
Last contact: Fri Feb 07 12:10:24 MSK 2014
Name: 10.10.1.14:50010
Decommission Status : Normal
Configured Capacity: 317070499840 (295.29 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 131537530880 (122.5 GB)
DFS Remaining: 185532944384(172.79 GB)
DFS Used%: 0%
DFS Remaining%: 58.51%
Last contact: Fri Feb 07 12:10:27 MSK 2014
Name: 10.10.1.11:50010
Decommission Status : Normal
Configured Capacity: 317070499840 (295.29 GB)
DFS Used: 1231216640 (1.15 GB)
Non DFS Used: 84698116096 (78.88 GB)
DFS Remaining: 231141167104(215.27 GB)
DFS Used%: 0.39%
DFS Remaining%: 72.9%
Last contact: Fri Feb 07 12:10:24 MSK 2014
Name: 10.10.1.16:50010
Decommission Status : Normal
Configured Capacity: 317070499840 (295.29 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 131537494016 (122.5 GB)
DFS Remaining: 185532981248(172.79 GB)
DFS Used%: 0%
DFS Remaining%: 58.51%
Last contact: Fri Feb 07 12:10:27 MSK 2014
Name: 10.10.1.12:50010
Decommission Status : Normal
Configured Capacity: 317070499840 (295.29 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 84642578432 (78.83 GB)
DFS Remaining: 232427896832(216.47 GB)
DFS Used%: 0%
DFS Remaining%: 73.3%
Last contact: Fri Feb 07 12:10:27 MSK 2014
Best,
Alex
Re: HDFS: file is not distributed after upload
Posted by Selçuk Şenkul <ss...@gmail.com>.
Hi Alex,
You should give the copyFromLocal command from the namenode or any machine
that is not a datanode to get the file distributed.
On Fri, Feb 7, 2014 at 10:53 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> The 0.20.203.0rc1 is a very old version at this point. Why not use a
> more current version if you're deploying a new cluster?
>
> Onto your issue, your configuration XML files (core-site.xml,
> hdfs-site.xml or mapred-site.xml) most likely have a "dfs.replication"
> value set to 1 causing only that may replicas to be written out by
> default.
>
> On Fri, Feb 7, 2014 at 2:11 PM, Alexander Frolov
> <al...@gmail.com> wrote:
> >
> > Hi, folks!
> >
> > I've deployed hadoop (0.20.203.0rc1) on 8-node cluster. After uploading
> file
> > onto hdfs I've got this file only on one of the nodes instead of being
> > uniformly distributed across all nodes. What can be the issue?
> >
> > $HADOOP_HOME/bin/hadoop dfs -copyFromLocal ../data/rmat-20.0
> > /user/frolo/input/rmat-20.0
> >
> > $HADOOP_HOME/bin/hadoop dfs -stat "%b %o %r %n" /user/frolo/input/rmat-*
> > 1220222968 67108864 1 rmat-20.0
> >
> > $HADOOP_HOME/bin/hadoop dfsadmin -report
> > Configured Capacity: 2536563998720 (2.31 TB)
> > Present Capacity: 1642543419392 (1.49 TB)
> > DFS Remaining: 1641312030720 (1.49 TB)
> > DFS Used: 1231388672 (1.15 GB)
> > DFS Used%: 0.07%
> > Under replicated blocks: 0
> > Blocks with corrupt replicas: 0
> > Missing blocks: 0
> >
> > -------------------------------------------------
> > Datanodes available: 8 (8 total, 0 dead)
> >
> > Name: 10.10.1.15:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131536928768 (122.5 GB)
> > DFS Remaining: 185533546496(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.13:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131533377536 (122.5 GB)
> > DFS Remaining: 185537097728(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.52%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.17:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 120023924736 (111.78 GB)
> > DFS Remaining: 197046550528(183.51 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 62.15%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.18:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 78510628864 (73.12 GB)
> > DFS Remaining: 238559846400(222.18 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 75.24%
> > Last contact: Fri Feb 07 12:10:24 MSK 2014
> >
> >
> > Name: 10.10.1.14:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131537530880 (122.5 GB)
> > DFS Remaining: 185532944384(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.11:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 1231216640 (1.15 GB)
> > Non DFS Used: 84698116096 (78.88 GB)
> > DFS Remaining: 231141167104(215.27 GB)
> > DFS Used%: 0.39%
> > DFS Remaining%: 72.9%
> > Last contact: Fri Feb 07 12:10:24 MSK 2014
> >
> >
> > Name: 10.10.1.16:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131537494016 (122.5 GB)
> > DFS Remaining: 185532981248(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.12:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 84642578432 (78.83 GB)
> > DFS Remaining: 232427896832(216.47 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 73.3%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Best,
> > Alex
> >
>
>
>
> --
> Harsh J
>
Re: HDFS: file is not distributed after upload
Posted by Selçuk Şenkul <ss...@gmail.com>.
Hi Alex,
You should give the copyFromLocal command from the namenode or any machine
that is not a datanode to get the file distributed.
On Fri, Feb 7, 2014 at 10:53 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> The 0.20.203.0rc1 is a very old version at this point. Why not use a
> more current version if you're deploying a new cluster?
>
> Onto your issue, your configuration XML files (core-site.xml,
> hdfs-site.xml or mapred-site.xml) most likely have a "dfs.replication"
> value set to 1 causing only that may replicas to be written out by
> default.
>
> On Fri, Feb 7, 2014 at 2:11 PM, Alexander Frolov
> <al...@gmail.com> wrote:
> >
> > Hi, folks!
> >
> > I've deployed hadoop (0.20.203.0rc1) on 8-node cluster. After uploading
> file
> > onto hdfs I've got this file only on one of the nodes instead of being
> > uniformly distributed across all nodes. What can be the issue?
> >
> > $HADOOP_HOME/bin/hadoop dfs -copyFromLocal ../data/rmat-20.0
> > /user/frolo/input/rmat-20.0
> >
> > $HADOOP_HOME/bin/hadoop dfs -stat "%b %o %r %n" /user/frolo/input/rmat-*
> > 1220222968 67108864 1 rmat-20.0
> >
> > $HADOOP_HOME/bin/hadoop dfsadmin -report
> > Configured Capacity: 2536563998720 (2.31 TB)
> > Present Capacity: 1642543419392 (1.49 TB)
> > DFS Remaining: 1641312030720 (1.49 TB)
> > DFS Used: 1231388672 (1.15 GB)
> > DFS Used%: 0.07%
> > Under replicated blocks: 0
> > Blocks with corrupt replicas: 0
> > Missing blocks: 0
> >
> > -------------------------------------------------
> > Datanodes available: 8 (8 total, 0 dead)
> >
> > Name: 10.10.1.15:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131536928768 (122.5 GB)
> > DFS Remaining: 185533546496(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.13:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131533377536 (122.5 GB)
> > DFS Remaining: 185537097728(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.52%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.17:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 120023924736 (111.78 GB)
> > DFS Remaining: 197046550528(183.51 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 62.15%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.18:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 78510628864 (73.12 GB)
> > DFS Remaining: 238559846400(222.18 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 75.24%
> > Last contact: Fri Feb 07 12:10:24 MSK 2014
> >
> >
> > Name: 10.10.1.14:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131537530880 (122.5 GB)
> > DFS Remaining: 185532944384(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.11:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 1231216640 (1.15 GB)
> > Non DFS Used: 84698116096 (78.88 GB)
> > DFS Remaining: 231141167104(215.27 GB)
> > DFS Used%: 0.39%
> > DFS Remaining%: 72.9%
> > Last contact: Fri Feb 07 12:10:24 MSK 2014
> >
> >
> > Name: 10.10.1.16:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131537494016 (122.5 GB)
> > DFS Remaining: 185532981248(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.12:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 84642578432 (78.83 GB)
> > DFS Remaining: 232427896832(216.47 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 73.3%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Best,
> > Alex
> >
>
>
>
> --
> Harsh J
>
Re: HDFS: file is not distributed after upload
Posted by Selçuk Şenkul <ss...@gmail.com>.
Hi Alex,
You should give the copyFromLocal command from the namenode or any machine
that is not a datanode to get the file distributed.
On Fri, Feb 7, 2014 at 10:53 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> The 0.20.203.0rc1 is a very old version at this point. Why not use a
> more current version if you're deploying a new cluster?
>
> Onto your issue, your configuration XML files (core-site.xml,
> hdfs-site.xml or mapred-site.xml) most likely have a "dfs.replication"
> value set to 1 causing only that may replicas to be written out by
> default.
>
> On Fri, Feb 7, 2014 at 2:11 PM, Alexander Frolov
> <al...@gmail.com> wrote:
> >
> > Hi, folks!
> >
> > I've deployed hadoop (0.20.203.0rc1) on 8-node cluster. After uploading
> file
> > onto hdfs I've got this file only on one of the nodes instead of being
> > uniformly distributed across all nodes. What can be the issue?
> >
> > $HADOOP_HOME/bin/hadoop dfs -copyFromLocal ../data/rmat-20.0
> > /user/frolo/input/rmat-20.0
> >
> > $HADOOP_HOME/bin/hadoop dfs -stat "%b %o %r %n" /user/frolo/input/rmat-*
> > 1220222968 67108864 1 rmat-20.0
> >
> > $HADOOP_HOME/bin/hadoop dfsadmin -report
> > Configured Capacity: 2536563998720 (2.31 TB)
> > Present Capacity: 1642543419392 (1.49 TB)
> > DFS Remaining: 1641312030720 (1.49 TB)
> > DFS Used: 1231388672 (1.15 GB)
> > DFS Used%: 0.07%
> > Under replicated blocks: 0
> > Blocks with corrupt replicas: 0
> > Missing blocks: 0
> >
> > -------------------------------------------------
> > Datanodes available: 8 (8 total, 0 dead)
> >
> > Name: 10.10.1.15:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131536928768 (122.5 GB)
> > DFS Remaining: 185533546496(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.13:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131533377536 (122.5 GB)
> > DFS Remaining: 185537097728(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.52%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.17:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 120023924736 (111.78 GB)
> > DFS Remaining: 197046550528(183.51 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 62.15%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.18:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 78510628864 (73.12 GB)
> > DFS Remaining: 238559846400(222.18 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 75.24%
> > Last contact: Fri Feb 07 12:10:24 MSK 2014
> >
> >
> > Name: 10.10.1.14:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131537530880 (122.5 GB)
> > DFS Remaining: 185532944384(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.11:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 1231216640 (1.15 GB)
> > Non DFS Used: 84698116096 (78.88 GB)
> > DFS Remaining: 231141167104(215.27 GB)
> > DFS Used%: 0.39%
> > DFS Remaining%: 72.9%
> > Last contact: Fri Feb 07 12:10:24 MSK 2014
> >
> >
> > Name: 10.10.1.16:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131537494016 (122.5 GB)
> > DFS Remaining: 185532981248(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.12:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 84642578432 (78.83 GB)
> > DFS Remaining: 232427896832(216.47 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 73.3%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Best,
> > Alex
> >
>
>
>
> --
> Harsh J
>
Re: HDFS: file is not distributed after upload
Posted by Selçuk Şenkul <ss...@gmail.com>.
Hi Alex,
You should give the copyFromLocal command from the namenode or any machine
that is not a datanode to get the file distributed.
On Fri, Feb 7, 2014 at 10:53 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> The 0.20.203.0rc1 is a very old version at this point. Why not use a
> more current version if you're deploying a new cluster?
>
> Onto your issue, your configuration XML files (core-site.xml,
> hdfs-site.xml or mapred-site.xml) most likely have a "dfs.replication"
> value set to 1 causing only that may replicas to be written out by
> default.
>
> On Fri, Feb 7, 2014 at 2:11 PM, Alexander Frolov
> <al...@gmail.com> wrote:
> >
> > Hi, folks!
> >
> > I've deployed hadoop (0.20.203.0rc1) on 8-node cluster. After uploading
> file
> > onto hdfs I've got this file only on one of the nodes instead of being
> > uniformly distributed across all nodes. What can be the issue?
> >
> > $HADOOP_HOME/bin/hadoop dfs -copyFromLocal ../data/rmat-20.0
> > /user/frolo/input/rmat-20.0
> >
> > $HADOOP_HOME/bin/hadoop dfs -stat "%b %o %r %n" /user/frolo/input/rmat-*
> > 1220222968 67108864 1 rmat-20.0
> >
> > $HADOOP_HOME/bin/hadoop dfsadmin -report
> > Configured Capacity: 2536563998720 (2.31 TB)
> > Present Capacity: 1642543419392 (1.49 TB)
> > DFS Remaining: 1641312030720 (1.49 TB)
> > DFS Used: 1231388672 (1.15 GB)
> > DFS Used%: 0.07%
> > Under replicated blocks: 0
> > Blocks with corrupt replicas: 0
> > Missing blocks: 0
> >
> > -------------------------------------------------
> > Datanodes available: 8 (8 total, 0 dead)
> >
> > Name: 10.10.1.15:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131536928768 (122.5 GB)
> > DFS Remaining: 185533546496(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.13:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131533377536 (122.5 GB)
> > DFS Remaining: 185537097728(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.52%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.17:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 120023924736 (111.78 GB)
> > DFS Remaining: 197046550528(183.51 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 62.15%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.18:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 78510628864 (73.12 GB)
> > DFS Remaining: 238559846400(222.18 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 75.24%
> > Last contact: Fri Feb 07 12:10:24 MSK 2014
> >
> >
> > Name: 10.10.1.14:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131537530880 (122.5 GB)
> > DFS Remaining: 185532944384(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.11:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 1231216640 (1.15 GB)
> > Non DFS Used: 84698116096 (78.88 GB)
> > DFS Remaining: 231141167104(215.27 GB)
> > DFS Used%: 0.39%
> > DFS Remaining%: 72.9%
> > Last contact: Fri Feb 07 12:10:24 MSK 2014
> >
> >
> > Name: 10.10.1.16:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131537494016 (122.5 GB)
> > DFS Remaining: 185532981248(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.12:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 84642578432 (78.83 GB)
> > DFS Remaining: 232427896832(216.47 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 73.3%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Best,
> > Alex
> >
>
>
>
> --
> Harsh J
>
Re: HDFS: file is not distributed after upload
Posted by Harsh J <ha...@cloudera.com>.
Hi,
The 0.20.203.0rc1 is a very old version at this point. Why not use a
more current version if you're deploying a new cluster?
Onto your issue, your configuration XML files (core-site.xml,
hdfs-site.xml or mapred-site.xml) most likely have a "dfs.replication"
value set to 1 causing only that may replicas to be written out by
default.
On Fri, Feb 7, 2014 at 2:11 PM, Alexander Frolov
<al...@gmail.com> wrote:
>
> Hi, folks!
>
> I've deployed hadoop (0.20.203.0rc1) on 8-node cluster. After uploading file
> onto hdfs I've got this file only on one of the nodes instead of being
> uniformly distributed across all nodes. What can be the issue?
>
> $HADOOP_HOME/bin/hadoop dfs -copyFromLocal ../data/rmat-20.0
> /user/frolo/input/rmat-20.0
>
> $HADOOP_HOME/bin/hadoop dfs -stat "%b %o %r %n" /user/frolo/input/rmat-*
> 1220222968 67108864 1 rmat-20.0
>
> $HADOOP_HOME/bin/hadoop dfsadmin -report
> Configured Capacity: 2536563998720 (2.31 TB)
> Present Capacity: 1642543419392 (1.49 TB)
> DFS Remaining: 1641312030720 (1.49 TB)
> DFS Used: 1231388672 (1.15 GB)
> DFS Used%: 0.07%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 8 (8 total, 0 dead)
>
> Name: 10.10.1.15:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131536928768 (122.5 GB)
> DFS Remaining: 185533546496(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.13:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131533377536 (122.5 GB)
> DFS Remaining: 185537097728(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.52%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.17:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 120023924736 (111.78 GB)
> DFS Remaining: 197046550528(183.51 GB)
> DFS Used%: 0%
> DFS Remaining%: 62.15%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.18:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 78510628864 (73.12 GB)
> DFS Remaining: 238559846400(222.18 GB)
> DFS Used%: 0%
> DFS Remaining%: 75.24%
> Last contact: Fri Feb 07 12:10:24 MSK 2014
>
>
> Name: 10.10.1.14:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131537530880 (122.5 GB)
> DFS Remaining: 185532944384(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.11:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 1231216640 (1.15 GB)
> Non DFS Used: 84698116096 (78.88 GB)
> DFS Remaining: 231141167104(215.27 GB)
> DFS Used%: 0.39%
> DFS Remaining%: 72.9%
> Last contact: Fri Feb 07 12:10:24 MSK 2014
>
>
> Name: 10.10.1.16:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131537494016 (122.5 GB)
> DFS Remaining: 185532981248(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.12:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 84642578432 (78.83 GB)
> DFS Remaining: 232427896832(216.47 GB)
> DFS Used%: 0%
> DFS Remaining%: 73.3%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Best,
> Alex
>
--
Harsh J
Re: HDFS: file is not distributed after upload
Posted by Harsh J <ha...@cloudera.com>.
Hi,
The 0.20.203.0rc1 is a very old version at this point. Why not use a
more current version if you're deploying a new cluster?
Onto your issue, your configuration XML files (core-site.xml,
hdfs-site.xml or mapred-site.xml) most likely have a "dfs.replication"
value set to 1 causing only that may replicas to be written out by
default.
On Fri, Feb 7, 2014 at 2:11 PM, Alexander Frolov
<al...@gmail.com> wrote:
>
> Hi, folks!
>
> I've deployed hadoop (0.20.203.0rc1) on 8-node cluster. After uploading file
> onto hdfs I've got this file only on one of the nodes instead of being
> uniformly distributed across all nodes. What can be the issue?
>
> $HADOOP_HOME/bin/hadoop dfs -copyFromLocal ../data/rmat-20.0
> /user/frolo/input/rmat-20.0
>
> $HADOOP_HOME/bin/hadoop dfs -stat "%b %o %r %n" /user/frolo/input/rmat-*
> 1220222968 67108864 1 rmat-20.0
>
> $HADOOP_HOME/bin/hadoop dfsadmin -report
> Configured Capacity: 2536563998720 (2.31 TB)
> Present Capacity: 1642543419392 (1.49 TB)
> DFS Remaining: 1641312030720 (1.49 TB)
> DFS Used: 1231388672 (1.15 GB)
> DFS Used%: 0.07%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 8 (8 total, 0 dead)
>
> Name: 10.10.1.15:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131536928768 (122.5 GB)
> DFS Remaining: 185533546496(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.13:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131533377536 (122.5 GB)
> DFS Remaining: 185537097728(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.52%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.17:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 120023924736 (111.78 GB)
> DFS Remaining: 197046550528(183.51 GB)
> DFS Used%: 0%
> DFS Remaining%: 62.15%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.18:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 78510628864 (73.12 GB)
> DFS Remaining: 238559846400(222.18 GB)
> DFS Used%: 0%
> DFS Remaining%: 75.24%
> Last contact: Fri Feb 07 12:10:24 MSK 2014
>
>
> Name: 10.10.1.14:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131537530880 (122.5 GB)
> DFS Remaining: 185532944384(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.11:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 1231216640 (1.15 GB)
> Non DFS Used: 84698116096 (78.88 GB)
> DFS Remaining: 231141167104(215.27 GB)
> DFS Used%: 0.39%
> DFS Remaining%: 72.9%
> Last contact: Fri Feb 07 12:10:24 MSK 2014
>
>
> Name: 10.10.1.16:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131537494016 (122.5 GB)
> DFS Remaining: 185532981248(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.12:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 84642578432 (78.83 GB)
> DFS Remaining: 232427896832(216.47 GB)
> DFS Used%: 0%
> DFS Remaining%: 73.3%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Best,
> Alex
>
--
Harsh J
Re: HDFS: file is not distributed after upload
Posted by Harsh J <ha...@cloudera.com>.
Hi,
The 0.20.203.0rc1 is a very old version at this point. Why not use a
more current version if you're deploying a new cluster?
Onto your issue, your configuration XML files (core-site.xml,
hdfs-site.xml or mapred-site.xml) most likely have a "dfs.replication"
value set to 1 causing only that may replicas to be written out by
default.
On Fri, Feb 7, 2014 at 2:11 PM, Alexander Frolov
<al...@gmail.com> wrote:
>
> Hi, folks!
>
> I've deployed hadoop (0.20.203.0rc1) on 8-node cluster. After uploading file
> onto hdfs I've got this file only on one of the nodes instead of being
> uniformly distributed across all nodes. What can be the issue?
>
> $HADOOP_HOME/bin/hadoop dfs -copyFromLocal ../data/rmat-20.0
> /user/frolo/input/rmat-20.0
>
> $HADOOP_HOME/bin/hadoop dfs -stat "%b %o %r %n" /user/frolo/input/rmat-*
> 1220222968 67108864 1 rmat-20.0
>
> $HADOOP_HOME/bin/hadoop dfsadmin -report
> Configured Capacity: 2536563998720 (2.31 TB)
> Present Capacity: 1642543419392 (1.49 TB)
> DFS Remaining: 1641312030720 (1.49 TB)
> DFS Used: 1231388672 (1.15 GB)
> DFS Used%: 0.07%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 8 (8 total, 0 dead)
>
> Name: 10.10.1.15:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131536928768 (122.5 GB)
> DFS Remaining: 185533546496(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.13:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131533377536 (122.5 GB)
> DFS Remaining: 185537097728(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.52%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.17:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 120023924736 (111.78 GB)
> DFS Remaining: 197046550528(183.51 GB)
> DFS Used%: 0%
> DFS Remaining%: 62.15%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.18:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 78510628864 (73.12 GB)
> DFS Remaining: 238559846400(222.18 GB)
> DFS Used%: 0%
> DFS Remaining%: 75.24%
> Last contact: Fri Feb 07 12:10:24 MSK 2014
>
>
> Name: 10.10.1.14:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131537530880 (122.5 GB)
> DFS Remaining: 185532944384(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.11:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 1231216640 (1.15 GB)
> Non DFS Used: 84698116096 (78.88 GB)
> DFS Remaining: 231141167104(215.27 GB)
> DFS Used%: 0.39%
> DFS Remaining%: 72.9%
> Last contact: Fri Feb 07 12:10:24 MSK 2014
>
>
> Name: 10.10.1.16:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131537494016 (122.5 GB)
> DFS Remaining: 185532981248(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.12:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 84642578432 (78.83 GB)
> DFS Remaining: 232427896832(216.47 GB)
> DFS Used%: 0%
> DFS Remaining%: 73.3%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Best,
> Alex
>
--
Harsh J
Re: HDFS: file is not distributed after upload
Posted by Harsh J <ha...@cloudera.com>.
Hi,
The 0.20.203.0rc1 is a very old version at this point. Why not use a
more current version if you're deploying a new cluster?
Onto your issue, your configuration XML files (core-site.xml,
hdfs-site.xml or mapred-site.xml) most likely have a "dfs.replication"
value set to 1 causing only that may replicas to be written out by
default.
On Fri, Feb 7, 2014 at 2:11 PM, Alexander Frolov
<al...@gmail.com> wrote:
>
> Hi, folks!
>
> I've deployed hadoop (0.20.203.0rc1) on 8-node cluster. After uploading file
> onto hdfs I've got this file only on one of the nodes instead of being
> uniformly distributed across all nodes. What can be the issue?
>
> $HADOOP_HOME/bin/hadoop dfs -copyFromLocal ../data/rmat-20.0
> /user/frolo/input/rmat-20.0
>
> $HADOOP_HOME/bin/hadoop dfs -stat "%b %o %r %n" /user/frolo/input/rmat-*
> 1220222968 67108864 1 rmat-20.0
>
> $HADOOP_HOME/bin/hadoop dfsadmin -report
> Configured Capacity: 2536563998720 (2.31 TB)
> Present Capacity: 1642543419392 (1.49 TB)
> DFS Remaining: 1641312030720 (1.49 TB)
> DFS Used: 1231388672 (1.15 GB)
> DFS Used%: 0.07%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 8 (8 total, 0 dead)
>
> Name: 10.10.1.15:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131536928768 (122.5 GB)
> DFS Remaining: 185533546496(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.13:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131533377536 (122.5 GB)
> DFS Remaining: 185537097728(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.52%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.17:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 120023924736 (111.78 GB)
> DFS Remaining: 197046550528(183.51 GB)
> DFS Used%: 0%
> DFS Remaining%: 62.15%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.18:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 78510628864 (73.12 GB)
> DFS Remaining: 238559846400(222.18 GB)
> DFS Used%: 0%
> DFS Remaining%: 75.24%
> Last contact: Fri Feb 07 12:10:24 MSK 2014
>
>
> Name: 10.10.1.14:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131537530880 (122.5 GB)
> DFS Remaining: 185532944384(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.11:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 1231216640 (1.15 GB)
> Non DFS Used: 84698116096 (78.88 GB)
> DFS Remaining: 231141167104(215.27 GB)
> DFS Used%: 0.39%
> DFS Remaining%: 72.9%
> Last contact: Fri Feb 07 12:10:24 MSK 2014
>
>
> Name: 10.10.1.16:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131537494016 (122.5 GB)
> DFS Remaining: 185532981248(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.12:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 84642578432 (78.83 GB)
> DFS Remaining: 232427896832(216.47 GB)
> DFS Used%: 0%
> DFS Remaining%: 73.3%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Best,
> Alex
>
--
Harsh J