You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Sriram Rao <sr...@gmail.com> on 2008/10/16 01:10:22 UTC
distcp logs
Hi,
when using distcp, we often find that the distcp logs are in the
target dir. root. for instance,
bin/hadoop dist src dest
We end up with: dest/_distcp_logs_...
Is there a way to (1) tell distcp to put the logs elsewhere and (2)
nuke the logs when done?
Sriram
Re: Why did I only get 2 live datanodes?
Posted by Mafish Liu <ma...@gmail.com>.
Hi, wei:
Did you have all your nodes have the same directory structures, the same
configuration files, the same hadoop data directory, and make sure you have
access to the data structure.
You may have connection problems, MAKE SURE your slave node can ssh to
master without password. And TURN OFF FIREWALL in your master node.
--
Mafish@gmail.com
Institute of Computing Technology, Chinese Academy of Sciences, Beijing.
Re: Why did I only get 2 live datanodes?
Posted by David Wei <we...@kingsoft.com>.
Almost desperate!
After configuring all the machines connecting each other with hostname,
only ONE node (master itself) still alive now!
Conditions as follows:
1. each machine can access others with SSH without needing password input
2. every machine has the same /etc/hosts file listing the name of machines
3. all the machine has the same masters/slaves files listing the name of
machines
4. inside hadoop-site.xml file, master had been configured to the right
machine name
5. when starting cluster from master machine, found some logs like this:
org.apache.hadoop.dfs.SafeModeException: Cannot delete
/tmp/hadoop-root/mapred/system. Name node is in safe mode.
The ratio of reported blocks 0.8750 has not reached the threshold
0.9990. Safe mode will be turned off automatically.
at org.apache.hadoop.dfs.FSNamesystem.deleteInternal(FSNamesystem.java:1494)
at org.apache.hadoop.dfs.FSNamesystem.delete(FSNamesystem.java:1466)
at org.apache.hadoop.dfs.NameNode.delete(NameNode.java:425)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
... ...
6. inside slaves's log, it shows that slaves can not reach the master:
STARTUP_MSG: build =
http://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r
694836; compiled by 'hadoopqa' on Fri Sep 12 23:29:35 UTC 2008
************************************************************/
2008-10-16 15:35:44,142 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /192.168.52.129:9000. Already tried 0 time(s).
... ....
after many times retry, it stoped.
we had already try to delete the folder of hadoop-root inside /tmp, but
it is the same for the problem!
Anybody can help?
Thx!
David Wei 写道:
> It seems that this is not the point. ;=(
> And in this case, my cluster is really easy to crash....
>
>
> СP 写道:
>
>> According to the cofiguration settings manaul in this website
>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
>> i think it sould be
>>
>> Master
>> Slave1
>> Slave2
>> Slave3
>>
>> in the slaves configuration
>> maybe you should give each IP a hostname for identification
>>
>> ----- Original Message -----
>> From: "David Wei" <we...@kingsoft.com>
>> To: "?P" <pi...@cht.com.tw>
>> Cc: <co...@hadoop.apache.org>
>> Sent: Thursday, October 16, 2008 11:41 AM
>> Subject: Re: Why did I only get 2 live datanodes?
>>
>>
>>
>>
>>> only the IPs of each node. For example:
>>> 192.168.52.129
>>> 192.168.49.40
>>> 192.168.55.104
>>> 192.168.49.148
>>>
>>> They are totally different in hardware conf., and is that matter?
>>>
>>> Thx!
>>>
>>> David
>>>
>>> ?P 写道:
>>>
>>>
>>>> What is your configuration in $HADOOP_HOME/conf/salves ?
>>>>
>>>> ----- Original Message -----
>>>> From: "David Wei" <we...@kingsoft.com>
>>>> To: <co...@hadoop.apache.org>
>>>> Sent: Thursday, October 16, 2008 11:05 AM
>>>> Subject: Why did I only get 2 live datanodes?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> We had installed hadoop on 4 machines, and one of them has been chosen
>>>>> to be master and also slaves. The rest of machines are configured as
>>>>> slaves. But it is strange that we can only see 2 live nodes on web
>>>>> UI:http://192.168.52.129:50070/dfshealth.jsp(master machine). When we
>>>>> try to refresh the page, we found that only 2 nodes are listed and one
>>>>> of them are continuing replaced by others.
>>>>>
>>>>> After using command:bin/hadoop dfsadmin -report, we got only 2 data
>>>>> nodes listed.
>>>>>
>>>>> Another thing is we found this in the log
>>>>> file(hadoop-root-namenode-datacenter5.log
>>>>> ---datacenter5 is the name of master):
>>>>> 2008-10-16 02:45:12,384 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>>>>> NameSystem.registerDatanode: node 192.168.49.148:50010 is replaced by
>>>>> 192.168.52.129:50010 with the same storageID
>>>>> DS-2140035130-127.0.0.1-50010-1223898963914
>>>>> .......
>>>>> 2008-10-16 02:45:12,432 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>>>>> NameSystem.registerDatanode: node 192.168.52.129:50010 is replaced by
>>>>> 192.168.55.104:50010 with the same storageID
>>>>> DS-2140035130-127.0.0.1-50010-1223898963914
>>>>> ........
>>>>> 2008-10-16 02:45:13,380 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>>>>> NameSystem.registerDatanode: node 192.168.55.104:50010 is replaced by
>>>>> 192.168.49.148:50010 with the same storageID
>>>>> DS-2140035130-127.0.0.1-50010-1223898963914
>>>>>
>>>>> Is there any problem with our configuration or maybe we just missed some
>>>>> options?
>>>>>
>>>>> BTW, my machines are total different in hardware, such as memory, disk
>>>>> and cpu.
>>>>>
>>>>> Thx!
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>
>>
>
>
>
>
Re: Why did I only get 2 live datanodes?
Posted by David Wei <we...@kingsoft.com>.
It seems that this is not the point. ;=(
And in this case, my cluster is really easy to crash....
СP 写道:
> According to the cofiguration settings manaul in this website
> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
> i think it sould be
>
> Master
> Slave1
> Slave2
> Slave3
>
> in the slaves configuration
> maybe you should give each IP a hostname for identification
>
> ----- Original Message -----
> From: "David Wei" <we...@kingsoft.com>
> To: "?P" <pi...@cht.com.tw>
> Cc: <co...@hadoop.apache.org>
> Sent: Thursday, October 16, 2008 11:41 AM
> Subject: Re: Why did I only get 2 live datanodes?
>
>
>
>> only the IPs of each node. For example:
>> 192.168.52.129
>> 192.168.49.40
>> 192.168.55.104
>> 192.168.49.148
>>
>> They are totally different in hardware conf., and is that matter?
>>
>> Thx!
>>
>> David
>>
>> ?P 写道:
>>
>>> What is your configuration in $HADOOP_HOME/conf/salves ?
>>>
>>> ----- Original Message -----
>>> From: "David Wei" <we...@kingsoft.com>
>>> To: <co...@hadoop.apache.org>
>>> Sent: Thursday, October 16, 2008 11:05 AM
>>> Subject: Why did I only get 2 live datanodes?
>>>
>>>
>>>
>>>
>>>> We had installed hadoop on 4 machines, and one of them has been chosen
>>>> to be master and also slaves. The rest of machines are configured as
>>>> slaves. But it is strange that we can only see 2 live nodes on web
>>>> UI:http://192.168.52.129:50070/dfshealth.jsp(master machine). When we
>>>> try to refresh the page, we found that only 2 nodes are listed and one
>>>> of them are continuing replaced by others.
>>>>
>>>> After using command:bin/hadoop dfsadmin -report, we got only 2 data
>>>> nodes listed.
>>>>
>>>> Another thing is we found this in the log
>>>> file(hadoop-root-namenode-datacenter5.log
>>>> ---datacenter5 is the name of master):
>>>> 2008-10-16 02:45:12,384 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>>>> NameSystem.registerDatanode: node 192.168.49.148:50010 is replaced by
>>>> 192.168.52.129:50010 with the same storageID
>>>> DS-2140035130-127.0.0.1-50010-1223898963914
>>>> .......
>>>> 2008-10-16 02:45:12,432 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>>>> NameSystem.registerDatanode: node 192.168.52.129:50010 is replaced by
>>>> 192.168.55.104:50010 with the same storageID
>>>> DS-2140035130-127.0.0.1-50010-1223898963914
>>>> ........
>>>> 2008-10-16 02:45:13,380 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>>>> NameSystem.registerDatanode: node 192.168.55.104:50010 is replaced by
>>>> 192.168.49.148:50010 with the same storageID
>>>> DS-2140035130-127.0.0.1-50010-1223898963914
>>>>
>>>> Is there any problem with our configuration or maybe we just missed some
>>>> options?
>>>>
>>>> BTW, my machines are total different in hardware, such as memory, disk
>>>> and cpu.
>>>>
>>>> Thx!
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>
>
Re: Why did I only get 2 live datanodes?
Posted by David Wei <we...@kingsoft.com>.
only the IPs of each node. For example:
192.168.52.129
192.168.49.40
192.168.55.104
192.168.49.148
They are totally different in hardware conf., and is that matter?
Thx!
David
?P 写道:
> What is your configuration in $HADOOP_HOME/conf/salves ?
>
> ----- Original Message -----
> From: "David Wei" <we...@kingsoft.com>
> To: <co...@hadoop.apache.org>
> Sent: Thursday, October 16, 2008 11:05 AM
> Subject: Why did I only get 2 live datanodes?
>
>
>
>> We had installed hadoop on 4 machines, and one of them has been chosen
>> to be master and also slaves. The rest of machines are configured as
>> slaves. But it is strange that we can only see 2 live nodes on web
>> UI:http://192.168.52.129:50070/dfshealth.jsp(master machine). When we
>> try to refresh the page, we found that only 2 nodes are listed and one
>> of them are continuing replaced by others.
>>
>> After using command:bin/hadoop dfsadmin -report, we got only 2 data
>> nodes listed.
>>
>> Another thing is we found this in the log
>> file(hadoop-root-namenode-datacenter5.log
>> ---datacenter5 is the name of master):
>> 2008-10-16 02:45:12,384 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>> NameSystem.registerDatanode: node 192.168.49.148:50010 is replaced by
>> 192.168.52.129:50010 with the same storageID
>> DS-2140035130-127.0.0.1-50010-1223898963914
>> .......
>> 2008-10-16 02:45:12,432 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>> NameSystem.registerDatanode: node 192.168.52.129:50010 is replaced by
>> 192.168.55.104:50010 with the same storageID
>> DS-2140035130-127.0.0.1-50010-1223898963914
>> ........
>> 2008-10-16 02:45:13,380 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
>> NameSystem.registerDatanode: node 192.168.55.104:50010 is replaced by
>> 192.168.49.148:50010 with the same storageID
>> DS-2140035130-127.0.0.1-50010-1223898963914
>>
>> Is there any problem with our configuration or maybe we just missed some
>> options?
>>
>> BTW, my machines are total different in hardware, such as memory, disk
>> and cpu.
>>
>> Thx!
>>
>> David
>>
>>
>>
>>
>>
>
>
Re: Why did I only get 2 live datanodes?
Posted by ?P <pi...@cht.com.tw>.
What is your configuration in $HADOOP_HOME/conf/salves ?
----- Original Message -----
From: "David Wei" <we...@kingsoft.com>
To: <co...@hadoop.apache.org>
Sent: Thursday, October 16, 2008 11:05 AM
Subject: Why did I only get 2 live datanodes?
> We had installed hadoop on 4 machines, and one of them has been chosen
> to be master and also slaves. The rest of machines are configured as
> slaves. But it is strange that we can only see 2 live nodes on web
> UI:http://192.168.52.129:50070/dfshealth.jsp(master machine). When we
> try to refresh the page, we found that only 2 nodes are listed and one
> of them are continuing replaced by others.
>
> After using command:bin/hadoop dfsadmin -report, we got only 2 data
> nodes listed.
>
> Another thing is we found this in the log
> file(hadoop-root-namenode-datacenter5.log
> ---datacenter5 is the name of master):
> 2008-10-16 02:45:12,384 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
> NameSystem.registerDatanode: node 192.168.49.148:50010 is replaced by
> 192.168.52.129:50010 with the same storageID
> DS-2140035130-127.0.0.1-50010-1223898963914
> .......
> 2008-10-16 02:45:12,432 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
> NameSystem.registerDatanode: node 192.168.52.129:50010 is replaced by
> 192.168.55.104:50010 with the same storageID
> DS-2140035130-127.0.0.1-50010-1223898963914
> ........
> 2008-10-16 02:45:13,380 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
> NameSystem.registerDatanode: node 192.168.55.104:50010 is replaced by
> 192.168.49.148:50010 with the same storageID
> DS-2140035130-127.0.0.1-50010-1223898963914
>
> Is there any problem with our configuration or maybe we just missed some
> options?
>
> BTW, my machines are total different in hardware, such as memory, disk
> and cpu.
>
> Thx!
>
> David
>
>
>
>
Why did I only get 2 live datanodes?
Posted by David Wei <we...@kingsoft.com>.
We had installed hadoop on 4 machines, and one of them has been chosen
to be master and also slaves. The rest of machines are configured as
slaves. But it is strange that we can only see 2 live nodes on web
UI:http://192.168.52.129:50070/dfshealth.jsp(master machine). When we
try to refresh the page, we found that only 2 nodes are listed and one
of them are continuing replaced by others.
After using command:bin/hadoop dfsadmin -report, we got only 2 data
nodes listed.
Another thing is we found this in the log
file(hadoop-root-namenode-datacenter5.log
---datacenter5 is the name of master):
2008-10-16 02:45:12,384 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.registerDatanode: node 192.168.49.148:50010 is replaced by
192.168.52.129:50010 with the same storageID
DS-2140035130-127.0.0.1-50010-1223898963914
.......
2008-10-16 02:45:12,432 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.registerDatanode: node 192.168.52.129:50010 is replaced by
192.168.55.104:50010 with the same storageID
DS-2140035130-127.0.0.1-50010-1223898963914
........
2008-10-16 02:45:13,380 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.registerDatanode: node 192.168.55.104:50010 is replaced by
192.168.49.148:50010 with the same storageID
DS-2140035130-127.0.0.1-50010-1223898963914
Is there any problem with our configuration or maybe we just missed some
options?
BTW, my machines are total different in hardware, such as memory, disk
and cpu.
Thx!
David
Re: distcp logs
Posted by Chris Douglas <ch...@yahoo-inc.com>.
-log <logdir>
-C
On Oct 15, 2008, at 4:10 PM, Sriram Rao wrote:
> Hi,
>
> when using distcp, we often find that the distcp logs are in the
> target dir. root. for instance,
> bin/hadoop dist src dest
>
> We end up with: dest/_distcp_logs_...
>
> Is there a way to (1) tell distcp to put the logs elsewhere and (2)
> nuke the logs when done?
>
> Sriram
Re: How to change number of mappers in Hadoop streaming?
Posted by Erik Holstad <er...@gmail.com>.
Hi Steve!
I you can pass -jobconf mapred.map.tasks=$MAPPERS -jobconf
mapred.reduce.tasks=$REDUCERS
to the streaming job to set the number of reducers and mappers.
Regards Erik
On Wed, Oct 15, 2008 at 4:25 PM, Steve Gao <st...@yahoo.com> wrote:
> Is there a way to change number of mappers in Hadoop streaming command
> line?
> I know I can change hadoop-default.xml:
>
> <property>
> <name>mapred.map.tasks</name>
> <value>10</value>
> <description>The default number of map tasks per job. Typically set
> to a prime several times greater than number of available hosts.
> Ignored when mapred.job.tracker is "local".
> </description>
> </property>
>
> But that's for all jobs. What if I just want each job has different
> NUM_OF_Mappers themselves? Thanks
>
>
>
>
>
Re: Can you specify the user on the map-reduce cluster in Hadoop
streaming
Posted by Allen Wittenauer <aw...@yahoo-inc.com>.
On 11/10/08 12:21 PM, "Rick Hangartner" <ha...@strands.com> wrote:
> But is there a proper way to allow developers to specify a <remote_username>
> they legitimately have access to on the cluster if it is not the same
> as the <local_username> of the account on their own machine they are
> using to submit a streaming job without setting HDFS permissions to
> 777?
There are ways that the Hadoop "security" as currently implemented can
be bypassed. If you really want to know how, that's probably better not
asked on a public list. ;)
But I'm curious as to your actual use case.
From what I can gather from your description, there are two possible
solutions, depending upon what you're looking to accomplish:
A) Turn off permissions
B) Create a group and make the output directory group writable
We use B a lot. We don't use A at all.
Can you specify the user on the map-reduce cluster in Hadoop streaming
Posted by Rick Hangartner <ha...@strands.com>.
Hi,
To make a Hadoop/MapReduce available for developers to experiment
with, we are setting up a cluster with Hadoop/MapReduce and a dataset,
and providing instructions how developers can use streaming to submit
jobs from their own machines.
For purposes of explanation here, we can assume each user has access
to their own login account on every node in the cluster and has a
"home" directory set up for them in the HDFS (e.g. Unix/Linux login:
<remote_username> on the nodes, and a directory /user/
<remote_username> on the HDFS with permissions 755).
What we've discovered is that if developers are working in the
<local_username> on his or her machine, and
<local_username>=<remote_username> of their account on the cluster,
streaming works just fine. If the <local_username> and
<remote_username> don't match, though, when when they specify "-input"
and "-output" should come from and go to files in the
<remote_username> directory on HDFS, they get an:
"org.apache.hadoop.fs.permission.AccessControlException:
org.apache.hadoop.fs.permission.AccessControlException: Permission
denied: user=<local_username>, access=WRITE,
inode="<remote_username>":<remote_username>:users:rwxr-xr-x"
exception.
Specifying "-jobconf user.name=<remote_username>" with the streaming
command on doesn't seem to help.
Of course, this all makes sense from a security viewpoint and from how
I think I understand HDFS derives permissions from the OS. But is
there a proper way to allow developers to specify a <remote_username>
they legitimately have access to on the cluster if it is not the same
as the <local_username> of the account on their own machine they are
using to submit a streaming job without setting HDFS permissions to
777? I've search the documentation and email list for this info and
perhaps have overlooked the answer to this question, apologies in
advance if I have.
Thanks.
Best regards,
Rick Hangartner
How to change number of mappers in Hadoop streaming?
Posted by Steve Gao <st...@yahoo.com>.
Is there a way to change number of mappers in Hadoop streaming command line?
I know I can change hadoop-default.xml:
<property>
<name>mapred.map.tasks</name>
<value>10</value>
<description>The default number of map tasks per job. Typically set
to a prime several times greater than number of available hosts.
Ignored when mapred.job.tracker is "local".
</description>
</property>
But that's for all jobs. What if I just want each job has different NUM_OF_Mappers themselves? Thanks
How to change number of mappers in Hadoop streaming?
Posted by Steve Gao <st...@yahoo.com>.
Is there a way to change number of mappers in Hadoop streaming command line?
I know I can change hadoop-default.xml:
<property>
<name>mapred.map.tasks</name>
<value>10</value>
<description>The default number of map tasks per job. Typically set
to a prime several times greater than number of available hosts.
Ignored when mapred.job.tracker is "local".
</description>
</property>
But that's for all jobs. What if I just want each job has different NUM_OF_Mappers themselves? Thanks