You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by A Df <ab...@yahoo.com> on 2011/08/13 01:19:02 UTC
Hadoop Cluster setup - no datanode
Hello Mates:
Thanks to everyone for their help so far. I have learnt a lot and have now done single and pseudo mode. I have a hadoop cluster but I ran jps on the master node and slave node but not all process are started
master:
22160 NameNode
22716 Jps
22458 JobTracker
slave:
32195 Jps
I also checked the logs and I see files for all the datanodes, jobtracker, namenode, secondarynamenode, and tasktracker. The tasktracker has one slave node log missing. The namenode formatted correctly. I set the values for below so I'm not sure if I need more. My cluster is 11 nodes (1 master, 10 slaves). I do not have permission to access root only my directory so hadoop is installed in there. I can ssh to the slaves properly.
* fs.default.name, dfs.name.dir, dfs.data.dir, mapred.job.tracker, mapred.system.dir
It also gave errors regarding:
* it cannot find the hadoop-daemon.sh file but I can see it
/home/my-user/hadoop-0.20.2_cluster/bin/hadoop-daemon.sh: line 40: cd: /home/my-user/hadoop-0.20.2_cluster/bin: No such file or directory
* it has the wrong path for the hadoop-config.sh so which parameter sets this field??
/home/my-user/hadoop-0.20.2_cluster/bin/hadoop-daemon.sh: line 42: /home/my-user/hadoop-0.20.2_cluster/hadoop-config.sh: No such file or directory
* not being able to create the log directory on the same slave node that doesn't have its tasktracker, which parameters should be used to set the log directory?
The same slave node which is giving problems also has:
Usage: hadoop-daemon.sh [--config <conf-dir>] [--hosts hostlistfile] (start|stop) <hadoop-command> <args...>
Thanks for your help.
Cheers,
Tamara
Re: Hadoop Cluster setup - no datanode
Posted by A Df <ab...@yahoo.com>.
Hello:
After doing some more changes, the report shows only one datanode but different from the ones that I selected. The problem seems to be the datanode but I am not sure why. Does hadoop need to run as root? What if a shared filesystem is used for the nodes, how can I specify that the datanodes use a local storage?
bin/hadoop dfsadmin -report
Configured Capacity: 1043282395136 (971.63 GB)
Present Capacity: 613049589760 (570.95 GB)
DFS Remaining: 613049565184 (570.95 GB)
DFS Used: 24576 (24 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)
Name: 161.74.12.36:50010
Decommission Status : Normal
Configured Capacity: 1043282395136 (971.63 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 430232805376 (400.69 GB)
DFS Remaining: 613049565184(570.95 GB)
DFS Used%: 0%
DFS Remaining%: 58.76%
Last contact: Sat Aug 13 04:04:24 BST 2011
Cheers,
A Df
>________________________________
>From: A Df <ab...@yahoo.com>
>To: "common-user@hadoop.apache.org" <co...@hadoop.apache.org>; Harsh J <ha...@cloudera.com>
>Sent: Saturday, 13 August 2011, 2:48
>Subject: Re: Hadoop Cluster setup - no datanode
>
>Hello:
>
>I did more test again but now I noticed that only 3 nodes have datanodes while the others do not. I ran the admin report tool and the result is below. Where do i configure the capacity?
>
> bin/hadoop dfsadmin -report
>
>
>Configured Capacity: 0 (0 KB)
>Present Capacity: 0 (0 KB)
>DFS Remaining: 0 (0 KB)
>DFS Used: 0 (0 KB)
>DFS Used%: �%
>Under replicated blocks: 0
>Blocks with corrupt replicas: 0
>Missing blocks: 0
>
>-------------------------------------------------
>Datanodes available: 1 (1 total, 0 dead)
>
>Name: 161.74.12.36:50010
>Decommission Status : Normal
>Configured Capacity: 0 (0 KB)
>DFS Used: 0 (0 KB)
>Non DFS Used: 0 (0 KB)
>DFS Remaining: 0(0 KB)
>DFS Used%: 100%
>DFS Remaining%: 0%
>Last contact: Sat Aug 13 02:39:39 BST 2011
>
>Thanks,
>A Df
>
>
>
>>________________________________
>>From: A Df <ab...@yahoo.com>
>>To: "common-user@hadoop.apache.org" <co...@hadoop.apache.org>; Harsh J <ha...@cloudera.com>
>>Sent: Saturday, 13 August 2011, 0:19
>>Subject: Hadoop Cluster setup - no datanode
>>
>>Hello Mates:
>>
>>Thanks to everyone for their help so far. I have learnt a lot and have now done single and pseudo mode. I have a hadoop cluster but I ran jps on the master node and slave node but not all process are started
>>
>>master:
>>22160 NameNode
>>22716 Jps
>>22458 JobTracker
>>
>>slave:
>>32195 Jps
>>
>>I also checked the logs and I see files for all the datanodes, jobtracker, namenode, secondarynamenode, and tasktracker. The tasktracker has one slave node log missing. The namenode formatted correctly. I set the values for below so I'm not sure if I need more. My cluster is 11 nodes (1 master, 10 slaves). I do not have permission to access root only my directory so hadoop is installed in there. I can ssh to the slaves properly.
>> * fs.default.name, dfs.name.dir, dfs.data.dir, mapred.job.tracker, mapred.system.dir
>>
>>
>>It also gave errors regarding:
>> * it cannot find the hadoop-daemon.sh file but I can see it
>>
>>/home/my-user/hadoop-0.20.2_cluster/bin/hadoop-daemon.sh: line 40: cd: /home/my-user/hadoop-0.20.2_cluster/bin: No such file or directory
>>
>> * it has the wrong path for the hadoop-config.sh so which parameter sets this field??
>>
>>/home/my-user/hadoop-0.20.2_cluster/bin/hadoop-daemon.sh: line 42: /home/my-user/hadoop-0.20.2_cluster/hadoop-config.sh: No such file or directory
>>
>> * not being able to create the log directory on the same slave node that doesn't have its tasktracker, which parameters should be used to set the log directory?
>>
>>The same slave node which is giving problems also has:
>> Usage: hadoop-daemon.sh [--config <conf-dir>] [--hosts hostlistfile] (start|stop) <hadoop-command> <args...>
>>
>>
>>Thanks for your help.
>>
>>Cheers,
>>Tamara
>>
>>
>>
>
>
Re: Hadoop Cluster setup - no datanode
Posted by A Df <ab...@yahoo.com>.
Hello:
I did more test again but now I noticed that only 3 nodes have datanodes while the others do not. I ran the admin report tool and the result is below. Where do i configure the capacity?
bin/hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: �%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)
Name: 161.74.12.36:50010
Decommission Status : Normal
Configured Capacity: 0 (0 KB)
DFS Used: 0 (0 KB)
Non DFS Used: 0 (0 KB)
DFS Remaining: 0(0 KB)
DFS Used%: 100%
DFS Remaining%: 0%
Last contact: Sat Aug 13 02:39:39 BST 2011
Thanks,
A Df
>________________________________
>From: A Df <ab...@yahoo.com>
>To: "common-user@hadoop.apache.org" <co...@hadoop.apache.org>; Harsh J <ha...@cloudera.com>
>Sent: Saturday, 13 August 2011, 0:19
>Subject: Hadoop Cluster setup - no datanode
>
>Hello Mates:
>
>Thanks to everyone for their help so far. I have learnt a lot and have now done single and pseudo mode. I have a hadoop cluster but I ran jps on the master node and slave node but not all process are started
>
>master:
>22160 NameNode
>22716 Jps
>22458 JobTracker
>
>slave:
>32195 Jps
>
>I also checked the logs and I see files for all the datanodes, jobtracker, namenode, secondarynamenode, and tasktracker. The tasktracker has one slave node log missing. The namenode formatted correctly. I set the values for below so I'm not sure if I need more. My cluster is 11 nodes (1 master, 10 slaves). I do not have permission to access root only my directory so hadoop is installed in there. I can ssh to the slaves properly.
> * fs.default.name, dfs.name.dir, dfs.data.dir, mapred.job.tracker, mapred.system.dir
>
>
>It also gave errors regarding:
> * it cannot find the hadoop-daemon.sh file but I can see it
>
>/home/my-user/hadoop-0.20.2_cluster/bin/hadoop-daemon.sh: line 40: cd: /home/my-user/hadoop-0.20.2_cluster/bin: No such file or directory
>
> * it has the wrong path for the hadoop-config.sh so which parameter sets this field??
>
>/home/my-user/hadoop-0.20.2_cluster/bin/hadoop-daemon.sh: line 42: /home/my-user/hadoop-0.20.2_cluster/hadoop-config.sh: No such file or directory
>
> * not being able to create the log directory on the same slave node that doesn't have its tasktracker, which parameters should be used to set the log directory?
>
>The same slave node which is giving problems also has:
> Usage: hadoop-daemon.sh [--config <conf-dir>] [--hosts hostlistfile] (start|stop) <hadoop-command> <args...>
>
>
>Thanks for your help.
>
>Cheers,
>Tamara
>
>
>