You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by David Wei <we...@kingsoft.com> on 2008/10/16 05:05:23 UTC
Why did I only get 2 live datanodes?
We had installed hadoop on 4 machines, and one of them has been chosen
to be master and also slaves. The rest of machines are configured as
slaves. But it is strange that we can only see 2 live nodes on web
UI:http://192.168.52.129:50070/dfshealth.jsp(master machine). When we
try to refresh the page, we found that only 2 nodes are listed and one
of them are continuing replaced by others.
After using command:bin/hadoop dfsadmin -report, we got only 2 data
nodes listed.
Another thing is we found this in the log
file(hadoop-root-namenode-datacenter5.log
---datacenter5 is the name of master):
2008-10-16 02:45:12,384 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.registerDatanode: node 192.168.49.148:50010 is replaced by
192.168.52.129:50010 with the same storageID
DS-2140035130-127.0.0.1-50010-1223898963914
.......
2008-10-16 02:45:12,432 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.registerDatanode: node 192.168.52.129:50010 is replaced by
192.168.55.104:50010 with the same storageID
DS-2140035130-127.0.0.1-50010-1223898963914
........
2008-10-16 02:45:13,380 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.registerDatanode: node 192.168.55.104:50010 is replaced by
192.168.49.148:50010 with the same storageID
DS-2140035130-127.0.0.1-50010-1223898963914
Is there any problem with our configuration or maybe we just missed some
options?
BTW, my machines are total different in hardware, such as memory, disk
and cpu.
Thx!
David
Re: Why did I only get 2 live datanodes?
Posted by Mafish Liu <ma...@gmail.com>.
Hi, wei:
Did you have all your nodes have the same directory structures, the same
configuration files, the same hadoop data directory, and make sure you have
access to the data structure.
You may have connection problems, MAKE SURE your slave node can ssh to
master without password. And TURN OFF FIREWALL in your master node.
--
Mafish@gmail.com
Institute of Computing Technology, Chinese Academy of Sciences, Beijing.
Re: Why did I only get 2 live datanodes?
Posted by David Wei <we...@kingsoft.com>.
Almost desperate!
After configuring all the machines connecting each other with hostname,
only ONE node (master itself) still alive now!
Conditions as follows:
1. each machine can access others with SSH without needing password input
2. every machine has the same /etc/hosts file listing the name of machines
3. all the machine has the same masters/slaves files listing the name of
machines
4. inside hadoop-site.xml file, master had been configured to the right
machine name
5. when starting cluster from master machine, found some logs like this:
org.apache.hadoop.dfs.SafeModeException: Cannot delete
/tmp/hadoop-root/mapred/system. Name node is in safe mode.
The ratio of reported blocks 0.8750 has not reached the threshold
0.9990. Safe mode will be turned off automatically.
at org.apache.hadoop.dfs.FSNamesystem.deleteInternal(FSNamesystem.java:1494)
at org.apache.hadoop.dfs.FSNamesystem.delete(FSNamesystem.java:1466)
at org.apache.hadoop.dfs.NameNode.delete(NameNode.java:425)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
... ...
6. inside slaves's log, it shows that slaves can not reach the master:
STARTUP_MSG: build =
http://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r
694836; compiled by 'hadoopqa' on Fri Sep 12 23:29:35 UTC 2008
************************************************************/
2008-10-16 15:35:44,142 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /192.168.52.129:9000. Already tried 0 time(s).
... ....
after many times retry, it stoped.
we had already try to delete the folder of hadoop-root inside /tmp, but
it is the same for the problem!
Anybody can help?
Thx!
David Wei 写道:
> It seems that this is not the point. ;=(
> And in this case, my cluster is really easy to crash....
>
>
> СP 写道:
>
>> According to the cofiguration settings manaul in this website
>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
>> i think it sould be
>>
>> Master
>> Slave1
>> Slave2
>> Slave3
>>
>> in the slaves configuration
>> maybe you should give each IP a hostname for identification
>>
>> ----- Original Message -----
>> From: "David Wei" <we...@kingsoft.com>
>> To: "?P" <pi...@cht.com.tw>
>> Cc: <co...@hadoop.apache.org>
>> Sent: Thursday, October 16, 2008 11:41 AM
>> Subject: Re: Why did I only get 2 live datanodes?
>>
>>
>>
>>
>>> only the IPs of each node. For example:
>>> 192.168.52.129
>>> 192.168.49.40
>>> 192.168.55.104
>>> 192.168.49.148
>>>
>>> They are totally different in hardware conf., and is that matter?
>>>
>>> Thx!
>>>
>>> David
>>>
>>> ?P 写道:
>>>
>>>
>>>> What is your configuration in $HADOOP_HOME/conf/salves ?
>>>>
>>>> ----- Original Message -----
>>>> From: "David Wei" <we...@kingsoft.com>
>>>> To: <co...@hadoop.apache.org>
>>>> Sent: Thursday, October 16, 2008 11:05 AM
>>>> Subject: Why did I only get 2 live datanodes?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> We had installed hadoop on 4 machines, and one of them has been chosen
>>>>> to be master and also slaves. The rest of machines are configured as
>>>>> slaves. But it is strange that we can only see 2 live nodes on web
>>>>> UI:http://192.168.52.129:50070/dfshealth.jsp(master machine). When we
>>>>> try to refresh the page, we found that only 2 nodes are listed and one
>>>>> of them are continuing replaced by others.
>>>>>
>>>>> After using command:bin/hadoop dfsadmin -report, we got only 2 data
>>>>> nodes listed.
>>>>>
>>>>> Another thing is we found this in the log
>>>>> file(hadoop-root-namenode-datacenter5.log
>>>>> ---datacenter5 is the name of master):
>>>>> 2008-10-16 02:45:12,384 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>>>>> NameSystem.registerDatanode: node 192.168.49.148:50010 is replaced by
>>>>> 192.168.52.129:50010 with the same storageID
>>>>> DS-2140035130-127.0.0.1-50010-1223898963914
>>>>> .......
>>>>> 2008-10-16 02:45:12,432 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>>>>> NameSystem.registerDatanode: node 192.168.52.129:50010 is replaced by
>>>>> 192.168.55.104:50010 with the same storageID
>>>>> DS-2140035130-127.0.0.1-50010-1223898963914
>>>>> ........
>>>>> 2008-10-16 02:45:13,380 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>>>>> NameSystem.registerDatanode: node 192.168.55.104:50010 is replaced by
>>>>> 192.168.49.148:50010 with the same storageID
>>>>> DS-2140035130-127.0.0.1-50010-1223898963914
>>>>>
>>>>> Is there any problem with our configuration or maybe we just missed some
>>>>> options?
>>>>>
>>>>> BTW, my machines are total different in hardware, such as memory, disk
>>>>> and cpu.
>>>>>
>>>>> Thx!
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>
>>
>
>
>
>
Re: Why did I only get 2 live datanodes?
Posted by David Wei <we...@kingsoft.com>.
It seems that this is not the point. ;=(
And in this case, my cluster is really easy to crash....
СP 写道:
> According to the cofiguration settings manaul in this website
> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
> i think it sould be
>
> Master
> Slave1
> Slave2
> Slave3
>
> in the slaves configuration
> maybe you should give each IP a hostname for identification
>
> ----- Original Message -----
> From: "David Wei" <we...@kingsoft.com>
> To: "?P" <pi...@cht.com.tw>
> Cc: <co...@hadoop.apache.org>
> Sent: Thursday, October 16, 2008 11:41 AM
> Subject: Re: Why did I only get 2 live datanodes?
>
>
>
>> only the IPs of each node. For example:
>> 192.168.52.129
>> 192.168.49.40
>> 192.168.55.104
>> 192.168.49.148
>>
>> They are totally different in hardware conf., and is that matter?
>>
>> Thx!
>>
>> David
>>
>> ?P 写道:
>>
>>> What is your configuration in $HADOOP_HOME/conf/salves ?
>>>
>>> ----- Original Message -----
>>> From: "David Wei" <we...@kingsoft.com>
>>> To: <co...@hadoop.apache.org>
>>> Sent: Thursday, October 16, 2008 11:05 AM
>>> Subject: Why did I only get 2 live datanodes?
>>>
>>>
>>>
>>>
>>>> We had installed hadoop on 4 machines, and one of them has been chosen
>>>> to be master and also slaves. The rest of machines are configured as
>>>> slaves. But it is strange that we can only see 2 live nodes on web
>>>> UI:http://192.168.52.129:50070/dfshealth.jsp(master machine). When we
>>>> try to refresh the page, we found that only 2 nodes are listed and one
>>>> of them are continuing replaced by others.
>>>>
>>>> After using command:bin/hadoop dfsadmin -report, we got only 2 data
>>>> nodes listed.
>>>>
>>>> Another thing is we found this in the log
>>>> file(hadoop-root-namenode-datacenter5.log
>>>> ---datacenter5 is the name of master):
>>>> 2008-10-16 02:45:12,384 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>>>> NameSystem.registerDatanode: node 192.168.49.148:50010 is replaced by
>>>> 192.168.52.129:50010 with the same storageID
>>>> DS-2140035130-127.0.0.1-50010-1223898963914
>>>> .......
>>>> 2008-10-16 02:45:12,432 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>>>> NameSystem.registerDatanode: node 192.168.52.129:50010 is replaced by
>>>> 192.168.55.104:50010 with the same storageID
>>>> DS-2140035130-127.0.0.1-50010-1223898963914
>>>> ........
>>>> 2008-10-16 02:45:13,380 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>>>> NameSystem.registerDatanode: node 192.168.55.104:50010 is replaced by
>>>> 192.168.49.148:50010 with the same storageID
>>>> DS-2140035130-127.0.0.1-50010-1223898963914
>>>>
>>>> Is there any problem with our configuration or maybe we just missed some
>>>> options?
>>>>
>>>> BTW, my machines are total different in hardware, such as memory, disk
>>>> and cpu.
>>>>
>>>> Thx!
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>
>
Re: Why did I only get 2 live datanodes?
Posted by David Wei <we...@kingsoft.com>.
only the IPs of each node. For example:
192.168.52.129
192.168.49.40
192.168.55.104
192.168.49.148
They are totally different in hardware conf., and is that matter?
Thx!
David
?P 写道:
> What is your configuration in $HADOOP_HOME/conf/salves ?
>
> ----- Original Message -----
> From: "David Wei" <we...@kingsoft.com>
> To: <co...@hadoop.apache.org>
> Sent: Thursday, October 16, 2008 11:05 AM
> Subject: Why did I only get 2 live datanodes?
>
>
>
>> We had installed hadoop on 4 machines, and one of them has been chosen
>> to be master and also slaves. The rest of machines are configured as
>> slaves. But it is strange that we can only see 2 live nodes on web
>> UI:http://192.168.52.129:50070/dfshealth.jsp(master machine). When we
>> try to refresh the page, we found that only 2 nodes are listed and one
>> of them are continuing replaced by others.
>>
>> After using command:bin/hadoop dfsadmin -report, we got only 2 data
>> nodes listed.
>>
>> Another thing is we found this in the log
>> file(hadoop-root-namenode-datacenter5.log
>> ---datacenter5 is the name of master):
>> 2008-10-16 02:45:12,384 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>> NameSystem.registerDatanode: node 192.168.49.148:50010 is replaced by
>> 192.168.52.129:50010 with the same storageID
>> DS-2140035130-127.0.0.1-50010-1223898963914
>> .......
>> 2008-10-16 02:45:12,432 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>> NameSystem.registerDatanode: node 192.168.52.129:50010 is replaced by
>> 192.168.55.104:50010 with the same storageID
>> DS-2140035130-127.0.0.1-50010-1223898963914
>> ........
>> 2008-10-16 02:45:13,380 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>> NameSystem.registerDatanode: node 192.168.55.104:50010 is replaced by
>> 192.168.49.148:50010 with the same storageID
>> DS-2140035130-127.0.0.1-50010-1223898963914
>>
>> Is there any problem with our configuration or maybe we just missed some
>> options?
>>
>> BTW, my machines are total different in hardware, such as memory, disk
>> and cpu.
>>
>> Thx!
>>
>> David
>>
>>
>>
>>
>>
>
>
Re: Why did I only get 2 live datanodes?
Posted by ?P <pi...@cht.com.tw>.
What is your configuration in $HADOOP_HOME/conf/salves ?
----- Original Message -----
From: "David Wei" <we...@kingsoft.com>
To: <co...@hadoop.apache.org>
Sent: Thursday, October 16, 2008 11:05 AM
Subject: Why did I only get 2 live datanodes?
> We had installed hadoop on 4 machines, and one of them has been chosen
> to be master and also slaves. The rest of machines are configured as
> slaves. But it is strange that we can only see 2 live nodes on web
> UI:http://192.168.52.129:50070/dfshealth.jsp(master machine). When we
> try to refresh the page, we found that only 2 nodes are listed and one
> of them are continuing replaced by others.
>
> After using command:bin/hadoop dfsadmin -report, we got only 2 data
> nodes listed.
>
> Another thing is we found this in the log
> file(hadoop-root-namenode-datacenter5.log
> ---datacenter5 is the name of master):
> 2008-10-16 02:45:12,384 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
> NameSystem.registerDatanode: node 192.168.49.148:50010 is replaced by
> 192.168.52.129:50010 with the same storageID
> DS-2140035130-127.0.0.1-50010-1223898963914
> .......
> 2008-10-16 02:45:12,432 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
> NameSystem.registerDatanode: node 192.168.52.129:50010 is replaced by
> 192.168.55.104:50010 with the same storageID
> DS-2140035130-127.0.0.1-50010-1223898963914
> ........
> 2008-10-16 02:45:13,380 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
> NameSystem.registerDatanode: node 192.168.55.104:50010 is replaced by
> 192.168.49.148:50010 with the same storageID
> DS-2140035130-127.0.0.1-50010-1223898963914
>
> Is there any problem with our configuration or maybe we just missed some
> options?
>
> BTW, my machines are total different in hardware, such as memory, disk
> and cpu.
>
> Thx!
>
> David
>
>
>
>