You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by jaylac <Ja...@cognizant.com> on 2007/03/23 04:42:31 UTC

System is hanging while executing bin/start-all.sh and bin/stop-all.sh

Hi

Whenever i execute the bin/start-all.sh command the slave node hangs.
Sometimes the master node hanga. If i restart the system and do the job,
then im getting the proper output... Same problem while stopping all the
daemons...

Have anyone faced this problem...

Please someone tell me the solution for this...

Im using two RED HAT LINUX machines... one master(10.229.62.6) and the other
slave(10.229.62.56)
In master node, the user name is jaya
In slave node, the user name is jaya

The steps which i follow is.....

Edit /home/jaya/.bashrc file
          Here ill set the HADOOP_CONF_DIR environment variable

MASTER NODE

1. Edit conf/slaves file....
        Contents
        ====================
         localhost
          jaya@10.229.62.56
         ====================

2. Edit conf/hadoop-en.sh file
      # The java implementation to use.  Required.
      export JAVA_HOME=/usr/java/jdk1.6.0

      # The maximum amount of heap to use, in MB. Default is 1000.
      export HADOOP_HEAPSIZE=200

      export HADOOP_HOME=/opt/hadoop-0.11.0

      # Extra Java runtime options.  Empty by default.
      export HADOOP_OPTS=-server
      
      # Where log files are stored.  $HADOOP_HOME/logs by default.
      export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

      # File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by
default.
      export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves

      Thats it.... No other changes in this file....

3. Edit conf/hadoop-site.xml file
       Contents
        ===========================================
         <?xml version="1.0"?>
         <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

         <!-- Put site-specific property overrides in this file. -->

         <configuration>

         <property>
         <name>fs.default.name</name>
         <value>10.229.62.6:50010</value>
         </property>

         <property>
         <name>mapred.job.tracker</name>
         <value>10.229.62.6:50011</value>
         </property>
         
         <property>
         <name>dfs.replication</name>
         <value>3</value>
         </property>
         
         </configuration>
         ====================================

 SLAVE NODE

1. Edit conf/masters file....
        Contents
        ====================
          jaya@10.229.62.56
         ====================

2. Edit conf/hadoop-en.sh file
      # The java implementation to use.  Required.
      export JAVA_HOME=/usr/java/jdk1.6.0

      # The maximum amount of heap to use, in MB. Default is 1000.
      export HADOOP_HEAPSIZE=200

      export HADOOP_HOME=/opt/hadoop-0.11.0

      # Extra Java runtime options.  Empty by default.
      export HADOOP_OPTS=-server
      
      # Where log files are stored.  $HADOOP_HOME/logs by default.
      export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

      # File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by
default.
      export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves

      Thats it.... No other changes in this file....

3. Edit conf/hadoop-site.xml file
       Contents
        ===========================================
         <?xml version="1.0"?>
         <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

         <!-- Put site-specific property overrides in this file. -->

         <configuration>

         <property>
         <name>fs.default.name</name>
         <value>10.229.62.6:50010</value>
         </property>

         <property>
         <name>mapred.job.tracker</name>
         <value>10.229.62.6:50011</value>
         </property>
         
         <property>
         <name>dfs.replication</name>
         <value>3</value>
         </property>
         
         </configuration>
         ====================================


I've already done steps for passwordless login

Thats is all........... Then ill perform the following operations....

In the HADOOP_HOME directory,

jaya@localhost hadoop-0.11.0]$ bin/hadoop namenode -format
Re-format filesystem in /tmp/hadoop-146736/dfs/name ? (Y or N) Y
Formatted /tmp/hadoop-146736/dfs/name
[jaya@localhost hadoop-0.11.0]$

Then

[jaya@localhost hadoop-0.11.0]$ bin/start-all.sh
tarting namenode, logging to
/opt/hadoop-0.11.0/logs/hadoop-jaya-namenode-local
host.localdomain.out
10.229.62.109: starting datanode, logging to
/opt/hadoop-0.11.0/logs/hadoop-jaya
-datanode-auriga.out
localhost: starting datanode, logging to
/opt/hadoop-0.11.0/logs/hadoop-jaya-dat
anode-localhost.localdomain.out
localhost: starting secondarynamenode, logging to
/opt/hadoop-0.11.0/logs/hadoop
-jaya-secondarynamenode-localhost.localdomain.out
starting jobtracker, logging to
/opt/hadoop-0.11.0/logs/hadoop-jaya-jobtracker-l
ocalhost.localdomain.out
10.229.62.109: starting tasktracker, logging to
/opt/hadoop-0.11.0/logs/hadoop-j
aya-tasktracker-auriga.out
localhost: starting tasktracker, logging to
/opt/hadoop-0.11.0/logs/hadoop-jaya-
tasktracker-localhost.localdomain.out

At this time the slave node gets hanged....

ill reastart the slave node....

Then im getting the proper output when i execute "bin/hadoop jar
hadoop-0.11.0-examples.jar input output"

Similarly, when i stop the daemons, the slave node is hanging... Sometimes
the master node is also hanging...

Please help me asap........

Thanks in advance 

Jaya
-- 
View this message in context: http://www.nabble.com/System-is-hanging-while-executing-bin-start-all.sh-and-bin-stop-all.sh-tf3451821.html#a9628679
Sent from the Hadoop Users mailing list archive at Nabble.com.


RE: System is hanging while executing bin/start-all.sh and bin/stop-all.sh

Posted by jaylac <Ja...@cognizant.com>.
Both the master and slave is Red hat Linux....

I cant understand the following line....
"one way to make sure this is only a local dns issue is to use ip addresses
in your setting.  if  it work, you can be sure it's only a local dns issue"



interesting.  i don't know how ssh can work togethor with dns, but other
program fail with dns...
what OS is auriga? 
one way to make sure this is only a local dns issue is to use ip addresses
in your setting.  if  it work, you can be sure it's only a local dns issue. 
and if this is the case, could you share your experiences with us?  thanks.

-- 
View this message in context: http://www.nabble.com/System-is-hanging-while-executing-bin-start-all.sh-and-bin-stop-all.sh-tf3451821.html#a9689520
Sent from the Hadoop Users mailing list archive at Nabble.com.


Re: System is hanging while executing bin/start-all.sh and bin/stop-all.sh

Posted by jaylac <Ja...@cognizant.com>.
[jaya@localhost ~]$ host auriga
Host auriga not found: 2(SERVFAIL)
[jaya@localhost ~]$     

But,
[jaya@localhost ~]$ ssh jaya@auriga is working



Arun C Murthy wrote:
> 
> jaylac wrote:
>> hi richard
>> 
>> My system is hanging always.. this is not a rare event for me.
>> I cant trace out what i the problem.... please help me...
>> 
>> Following are log files...
>> 
>> Here auriga is the name of thre slave node.
>> 
>> In the slave node
>> 
>> 2007-03-23 14:37:56,108 ERROR org.apache.hadoop.mapred.TaskTracker: Can
>> not
>> start task trackerbecause java.net.UnknownHostException: auriga: auriga
>>         at java.net.InetAddress.getLocalHost(InetAddress.java:1315)
>>         at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:169)
>> =============================================================================
>> 
>> hadoop-jaya-datanode-auriga.log file
>> ---------------------------------------------------------
>> 2007-03-23 14:37:54,248 ERROR org.apache.hadoop.dfs.DataNode:
>> java.net.UnknownHostException: auriga: auriga
>>         at java.net.InetAddress.getLocalHost(InetAddress.java:1315)
>>         at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:178)
> 
> It looks like a dns issue; please check with your sysadmin.
> 
> What does
> $ host auriga
> say?
> 
> Arun
> 
> 

-- 
View this message in context: http://www.nabble.com/System-is-hanging-while-executing-bin-start-all.sh-and-bin-stop-all.sh-tf3451821.html#a9686861
Sent from the Hadoop Users mailing list archive at Nabble.com.


RE: System is hanging while executing bin/start-all.sh and bin/stop-all.sh

Posted by jaylac <Ja...@cognizant.com>.
M$> ssh jaya@auriga is working without asking password.....



Richard Yang-3 wrote:
> 
> Hello,
> 
> It is nice to know the issue occurring consistently. 
> According to the log file, the namenode/master (M) has problem reaching to
> datanode/slave (S) auriga.  Does ssh from M to S work? i.e. does
> M$> ssh jaya@auriga
> directly return a command prompt on S as user jaya? If not, at least, we
> know there is some issue on network side.  One like Arun indicating, dns
> service is not functioning normally.
> One simple (coward) solution is to use only ip addresses.  This way could
> simplify the setup configurations.
> 
> 
> Best Regards
>  
> Richard Yang
> richardyang@richardyang.net
> kusanagiyang@gmail.com
>  
>  
> 
> -----Original Message-----
> From: Arun C Murthy [mailto:arunc@yahoo-inc.com] 
> Sent: Friday, March 23, 2007 3:33 AM
> To: hadoop-user@lucene.apache.org
> Subject: Re: System is hanging while executing bin/start-all.sh and
> bin/stop-all.sh
> 
> jaylac wrote:
>> hi richard
>> 
>> My system is hanging always.. this is not a rare event for me.
>> I cant trace out what i the problem.... please help me...
>> 
>> Following are log files...
>> 
>> Here auriga is the name of thre slave node.
>> 
>> In the slave node
>> 
>> 2007-03-23 14:37:56,108 ERROR org.apache.hadoop.mapred.TaskTracker: Can
> not
>> start task trackerbecause java.net.UnknownHostException: auriga: auriga
>>         at java.net.InetAddress.getLocalHost(InetAddress.java:1315)
>>         at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:169)
>>
> ============================================================================
> =
>> 
>> hadoop-jaya-datanode-auriga.log file
>> ---------------------------------------------------------
>> 2007-03-23 14:37:54,248 ERROR org.apache.hadoop.dfs.DataNode:
>> java.net.UnknownHostException: auriga: auriga
>>         at java.net.InetAddress.getLocalHost(InetAddress.java:1315)
>>         at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:178)
> 
> It looks like a dns issue; please check with your sysadmin.
> 
> What does
> $ host auriga
> say?
> 
> Arun
> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/System-is-hanging-while-executing-bin-start-all.sh-and-bin-stop-all.sh-tf3451821.html#a9686857
Sent from the Hadoop Users mailing list archive at Nabble.com.


RE: System is hanging while executing bin/start-all.sh and bin/stop-all.sh

Posted by Richard Yang <ri...@richardyang.net>.
Hello,

It is nice to know the issue occurring consistently. 
According to the log file, the namenode/master (M) has problem reaching to
datanode/slave (S) auriga.  Does ssh from M to S work? i.e. does
M$> ssh jaya@auriga
directly return a command prompt on S as user jaya? If not, at least, we
know there is some issue on network side.  One like Arun indicating, dns
service is not functioning normally.
One simple (coward) solution is to use only ip addresses.  This way could
simplify the setup configurations.


Best Regards
 
Richard Yang
richardyang@richardyang.net
kusanagiyang@gmail.com
 
 

-----Original Message-----
From: Arun C Murthy [mailto:arunc@yahoo-inc.com] 
Sent: Friday, March 23, 2007 3:33 AM
To: hadoop-user@lucene.apache.org
Subject: Re: System is hanging while executing bin/start-all.sh and
bin/stop-all.sh

jaylac wrote:
> hi richard
> 
> My system is hanging always.. this is not a rare event for me.
> I cant trace out what i the problem.... please help me...
> 
> Following are log files...
> 
> Here auriga is the name of thre slave node.
> 
> In the slave node
> 
> 2007-03-23 14:37:56,108 ERROR org.apache.hadoop.mapred.TaskTracker: Can
not
> start task trackerbecause java.net.UnknownHostException: auriga: auriga
>         at java.net.InetAddress.getLocalHost(InetAddress.java:1315)
>         at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:169)
>
============================================================================
=
> 
> hadoop-jaya-datanode-auriga.log file
> ---------------------------------------------------------
> 2007-03-23 14:37:54,248 ERROR org.apache.hadoop.dfs.DataNode:
> java.net.UnknownHostException: auriga: auriga
>         at java.net.InetAddress.getLocalHost(InetAddress.java:1315)
>         at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:178)

It looks like a dns issue; please check with your sysadmin.

What does
$ host auriga
say?

Arun



Re: System is hanging while executing bin/start-all.sh and bin/stop-all.sh

Posted by Arun C Murthy <ar...@yahoo-inc.com>.
jaylac wrote:
> hi richard
> 
> My system is hanging always.. this is not a rare event for me.
> I cant trace out what i the problem.... please help me...
> 
> Following are log files...
> 
> Here auriga is the name of thre slave node.
> 
> In the slave node
> 
> 2007-03-23 14:37:56,108 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
> start task trackerbecause java.net.UnknownHostException: auriga: auriga
>         at java.net.InetAddress.getLocalHost(InetAddress.java:1315)
>         at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:169)
> =============================================================================
> 
> hadoop-jaya-datanode-auriga.log file
> ---------------------------------------------------------
> 2007-03-23 14:37:54,248 ERROR org.apache.hadoop.dfs.DataNode:
> java.net.UnknownHostException: auriga: auriga
>         at java.net.InetAddress.getLocalHost(InetAddress.java:1315)
>         at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:178)

It looks like a dns issue; please check with your sysadmin.

What does
$ host auriga
say?

Arun

RE: System is hanging while executing bin/start-all.sh and bin/stop-all.sh

Posted by jaylac <Ja...@cognizant.com>.
hi richard

My system is hanging always.. this is not a rare event for me.
I cant trace out what i the problem.... please help me...

Following are log files...

Here auriga is the name of thre slave node.

In the slave node

==========================================================================
hadoop-jaya-tasktracker-auriga.log file
----------------------------------------------------------------

007-03-23 14:37:54,435 INFO org.mortbay.util.Credential: Checking Resource
aliases
2007-03-23 14:37:54,558 INFO org.mortbay.http.HttpServer: Version
Jetty/5.1.4
2007-03-23 14:37:54,570 INFO org.mortbay.util.Container: Started
HttpContext[/static,/static]
2007-03-23 14:37:54,570 INFO org.mortbay.util.Container: Started
HttpContext[/logs,/logs]
2007-03-23 14:37:55,949 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.WebApplicationHandler@3fa5ac
2007-03-23 14:37:56,047 INFO org.mortbay.util.Container: Started
WebApplicationContext[/,/]
2007-03-23 14:37:56,069 INFO org.mortbay.http.SocketListener: Started
SocketListener on 0.0.0.0:50060
2007-03-23 14:37:56,069 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.Server@a4e2e3
2007-03-23 14:37:56,108 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
start task trackerbecause java.net.UnknownHostException: auriga: auriga
        at java.net.InetAddress.getLocalHost(InetAddress.java:1315)
        at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:169)
        at
org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:247)
        at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:456)
        at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1499)
=============================================================================

hadoop-jaya-datanode-auriga.log file
---------------------------------------------------------
2007-03-23 14:37:54,248 ERROR org.apache.hadoop.dfs.DataNode:
java.net.UnknownHostException: auriga: auriga
        at java.net.InetAddress.getLocalHost(InetAddress.java:1315)
        at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:178)
        at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:1133)
        at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1061)
        at org.apache.hadoop.dfs.DataNode.runAndWait(DataNode.java:1091)
        at org.apache.hadoop.dfs.DataNode.main(DataNode.java:1255)



Hello Jaya,

It would be useful to check the namenode and datanode log file.  From my
past experiences, sometimes I would get java.io... kind of exceptions. I
think it could be relating to me using vmware. Though sometimes,
namenode/master simply could not connect to datanode/slaves (showing as some
ipc error message). I did the same thing to start the namenode/master and
the cluster is up and healthy again.  Too bad this phenomenon is not
consistent so I couldn't pin point how or why this happened.  Would you let
me know if you find the cause?  Thanks.

Best Regards
 
Richard Yang
richardyang@richardyang.net
kusanagiyang@gmail.com
-- 
View this message in context: http://www.nabble.com/System-is-hanging-while-executing-bin-start-all.sh-and-bin-stop-all.sh-tf3451821.html#a9631459
Sent from the Hadoop Users mailing list archive at Nabble.com.


RE: System is hanging while executing bin/start-all.sh and bin/stop-all.sh

Posted by Richard Yang <ri...@richardyang.net>.
Hello Jaya,

It would be useful to check the namenode and datanode log file.  From my
past experiences, sometimes I would get java.io... kind of exceptions. I
think it could be relating to me using vmware. Though sometimes,
namenode/master simply could not connect to datanode/slaves (showing as some
ipc error message). I did the same thing to start the namenode/master and
the cluster is up and healthy again.  Too bad this phenomenon is not
consistent so I couldn't pin point how or why this happened.  Would you let
me know if you find the cause?  Thanks.

Best Regards
 
Richard Yang
richardyang@richardyang.net
kusanagiyang@gmail.com
 
 
-----Original Message-----
From: jaylac [mailto:Jayalakshmi.Muniasamy@cognizant.com] 
Sent: Thursday, March 22, 2007 8:43 PM
To: hadoop-user@lucene.apache.org
Subject: System is hanging while executing bin/start-all.sh and
bin/stop-all.sh


Hi

Whenever i execute the bin/start-all.sh command the slave node hangs.
Sometimes the master node hanga. If i restart the system and do the job,
then im getting the proper output... Same problem while stopping all the
daemons...

Have anyone faced this problem...

Please someone tell me the solution for this...

Im using two RED HAT LINUX machines... one master(10.229.62.6) and the other
slave(10.229.62.56)
In master node, the user name is jaya
In slave node, the user name is jaya

The steps which i follow is.....

Edit /home/jaya/.bashrc file
          Here ill set the HADOOP_CONF_DIR environment variable

MASTER NODE

1. Edit conf/slaves file....
        Contents
        ====================
         localhost
          jaya@10.229.62.56
         ====================

2. Edit conf/hadoop-en.sh file
      # The java implementation to use.  Required.
      export JAVA_HOME=/usr/java/jdk1.6.0

      # The maximum amount of heap to use, in MB. Default is 1000.
      export HADOOP_HEAPSIZE=200

      export HADOOP_HOME=/opt/hadoop-0.11.0

      # Extra Java runtime options.  Empty by default.
      export HADOOP_OPTS=-server
      
      # Where log files are stored.  $HADOOP_HOME/logs by default.
      export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

      # File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by
default.
      export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves

      Thats it.... No other changes in this file....

3. Edit conf/hadoop-site.xml file
       Contents
        ===========================================
         <?xml version="1.0"?>
         <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

         <!-- Put site-specific property overrides in this file. -->

         <configuration>

         <property>
         <name>fs.default.name</name>
         <value>10.229.62.6:50010</value>
         </property>

         <property>
         <name>mapred.job.tracker</name>
         <value>10.229.62.6:50011</value>
         </property>
         
         <property>
         <name>dfs.replication</name>
         <value>3</value>
         </property>
         
         </configuration>
         ====================================

 SLAVE NODE

1. Edit conf/masters file....
        Contents
        ====================
          jaya@10.229.62.56
         ====================

2. Edit conf/hadoop-en.sh file
      # The java implementation to use.  Required.
      export JAVA_HOME=/usr/java/jdk1.6.0

      # The maximum amount of heap to use, in MB. Default is 1000.
      export HADOOP_HEAPSIZE=200

      export HADOOP_HOME=/opt/hadoop-0.11.0

      # Extra Java runtime options.  Empty by default.
      export HADOOP_OPTS=-server
      
      # Where log files are stored.  $HADOOP_HOME/logs by default.
      export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

      # File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by
default.
      export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves

      Thats it.... No other changes in this file....

3. Edit conf/hadoop-site.xml file
       Contents
        ===========================================
         <?xml version="1.0"?>
         <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

         <!-- Put site-specific property overrides in this file. -->

         <configuration>

         <property>
         <name>fs.default.name</name>
         <value>10.229.62.6:50010</value>
         </property>

         <property>
         <name>mapred.job.tracker</name>
         <value>10.229.62.6:50011</value>
         </property>
         
         <property>
         <name>dfs.replication</name>
         <value>3</value>
         </property>
         
         </configuration>
         ====================================


I've already done steps for passwordless login

Thats is all........... Then ill perform the following operations....

In the HADOOP_HOME directory,

jaya@localhost hadoop-0.11.0]$ bin/hadoop namenode -format
Re-format filesystem in /tmp/hadoop-146736/dfs/name ? (Y or N) Y
Formatted /tmp/hadoop-146736/dfs/name
[jaya@localhost hadoop-0.11.0]$

Then

[jaya@localhost hadoop-0.11.0]$ bin/start-all.sh
tarting namenode, logging to
/opt/hadoop-0.11.0/logs/hadoop-jaya-namenode-local
host.localdomain.out
10.229.62.109: starting datanode, logging to
/opt/hadoop-0.11.0/logs/hadoop-jaya
-datanode-auriga.out
localhost: starting datanode, logging to
/opt/hadoop-0.11.0/logs/hadoop-jaya-dat
anode-localhost.localdomain.out
localhost: starting secondarynamenode, logging to
/opt/hadoop-0.11.0/logs/hadoop
-jaya-secondarynamenode-localhost.localdomain.out
starting jobtracker, logging to
/opt/hadoop-0.11.0/logs/hadoop-jaya-jobtracker-l
ocalhost.localdomain.out
10.229.62.109: starting tasktracker, logging to
/opt/hadoop-0.11.0/logs/hadoop-j
aya-tasktracker-auriga.out
localhost: starting tasktracker, logging to
/opt/hadoop-0.11.0/logs/hadoop-jaya-
tasktracker-localhost.localdomain.out

At this time the slave node gets hanged....

ill reastart the slave node....

Then im getting the proper output when i execute "bin/hadoop jar
hadoop-0.11.0-examples.jar input output"

Similarly, when i stop the daemons, the slave node is hanging... Sometimes
the master node is also hanging...

Please help me asap........

Thanks in advance 

Jaya
-- 
View this message in context:
http://www.nabble.com/System-is-hanging-while-executing-bin-start-all.sh-and
-bin-stop-all.sh-tf3451821.html#a9628679
Sent from the Hadoop Users mailing list archive at Nabble.com.