You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by kladik <kl...@hotmail.com> on 2010/08/12 16:36:11 UTC
Configuring Hadoop Cluster
Hej,
I am configuring a Hadoop cluster with 1 master
(jobtracker+namenode+secondary namenode) and 2 slaves
(tasktracker+datanode). The 2 datanodes logs do not show any information but
the datanodes processes are running (checked with ps aux).
INITIALIZATION SCRIPT
bin/hadoop namenode -format
bin/start-all
The jobtracker WUI shows always state INITIALIZING with 0 nodes. However, if
I set only the master as a slave the cluster starts and runs successfully.
Can anybody give me any idea of what could be wrong? I attach the following
configuration files:
conf/master
master
conf/slaves
slave1
slave2
conf/core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000/</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>
</configuration>
conf/hdfs-site.xml
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/nutch/tmp/name/</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/nutch/tmp/data/</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
<description>Default block replication.
The actual number of replications can be specified when the file is
created.
The default is used if replication is not specified in create time.
</description>
</property>
</configuration>
conf/mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://master:9001</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
<property>
<name>mapred.map.tasks</name>
<value>4</value>
<description>
define mapred.map tasks to be number of slave hosts
</description>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>2</value>
<description>
define mapred.reduce tasks to be number of slave hosts
</description>
</property>
<property>
<name>mapred.system.dir</name>
<value>/nutch/tmp/mapreduce/system/</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/nutch/tmp/mapreduce/local/</value>
</property>
</configuration>
Thanks in advance ;)
--
View this message in context: http://lucene.472066.n3.nabble.com/Configuring-Hadoop-Cluster-tp1111545p1111545.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
RE: Configuring Hadoop Cluster
Posted by kladik <kl...@hotmail.com>.
thanks Kelvin, but it haven't worked and my datanode log files are empty yet.
--
View this message in context: http://lucene.472066.n3.nabble.com/Configuring-Hadoop-Cluster-tp1111545p1127800.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
RE: Configuring Hadoop Cluster
Posted by kladik <kl...@hotmail.com>.
Sorry sir, you were right.
First i had to change my /etc/sysconfig/iptables file in the namenode
computer, then i followed your instructions and...
my cluster is RUNNING!!!!
Thanks a lot, that was driving me crazy.
--
View this message in context: http://lucene.472066.n3.nabble.com/Configuring-Hadoop-Cluster-tp1111545p1129105.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
RE: Configuring Hadoop Cluster
Posted by Kelvin Rawls <ke...@iswcorp.com>.
Be sure firewall is off OR configured to allow access between master/lsave1/slave2
I use this on Ubuntu 10.04:
#!/bin/bash
# /sbin/iptables -X
# /sbin/iptables -t nat -F
# /sbin/iptables -t nat -X
# /sbin/iptables -t mangle -F
# /sbin/iptables -t mangle -X
/sbin/iptables -P INPUT ACCEPT
/sbin/iptables -P FORWARD ACCEPT
/sbin/iptables -P OUTPUT ACCEPT
/sbin/iptables -F
Kelvin
++++++++++++
Hej,
I am configuring a Hadoop cluster with 1 master
(jobtracker+namenode+secondary namenode) and 2 slaves
(tasktracker+datanode). The 2 datanodes logs do not show any information but
the datanodes processes are running (checked with ps aux).
INITIALIZATION SCRIPT
bin/hadoop namenode -format
bin/start-all
The jobtracker WUI shows always state INITIALIZING with 0 nodes. However, if
I set only the master as a slave the cluster starts and runs successfully.
Can anybody give me any idea of what could be wrong? I attach the following
configuration files:
conf/master
master
conf/slaves
slave1
slave2
conf/core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000/</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>
</configuration>
conf/hdfs-site.xml
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/nutch/tmp/name/</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/nutch/tmp/data/</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
<description>Default block replication.
The actual number of replications can be specified when the file is
created.
The default is used if replication is not specified in create time.
</description>
</property>
</configuration>
conf/mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://master:9001</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
<property>
<name>mapred.map.tasks</name>
<value>4</value>
<description>
define mapred.map tasks to be number of slave hosts
</description>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>2</value>
<description>
define mapred.reduce tasks to be number of slave hosts
</description>
</property>
<property>
<name>mapred.system.dir</name>
<value>/nutch/tmp/mapreduce/system/</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/nutch/tmp/mapreduce/local/</value>
</property>
</configuration>
Thanks in advance ;)
--
View this message in context: http://lucene.472066.n3.nabble.com/Configuring-Hadoop-Cluster-tp1111545p1111545.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.