You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Marianne Spiller <sp...@spring.de> on 2008/05/20 11:21:11 UTC
Node stops working
Hi,
I have several problems with a cluster running hadoop-0.16.0 and
java-1.6.0_02 (same setup on each machine, CentOS 5); at the moment, 18
of 20 machines are running -- load on each between 4 and 10. Two nodes
aren't working anymore, worker247 ([1] & [2]) and worker236 ([3] & [4]).
The error messages I get are incomprehensible to me; I double-checked
network settings, DNS configuration, ... I have no starting-point for
debugging :(
Can anyone please give me a hint?
Thanks in advance,
ms
[1] hadoop-jobtracker-worker247.log
2008-05-15 12:51:54,665 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = worker247/192.168.4.247
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.16.0
STARTUP_MSG: build =
http://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.16 -r
618351; compiled by
'hadoopqa' on Mon Feb 4 19:29:11 UTC 2008
************************************************************/
2008-05-15 12:51:55,077 FATAL org.apache.hadoop.mapred.JobTracker:
java.net.BindException: Problem binding to
master01/192.168.4.1:9001
at org.apache.hadoop.ipc.Server.bind(Server.java:193)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:252)
at org.apache.hadoop.ipc.Server.<init>(Server.java:973)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:393)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:355)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:639)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:124)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:2114)
2008-05-15 12:51:55,078 INFO org.apache.hadoop.mapred.JobTracker:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at worker247/192.168.4.247
************************************************************/
[2] hadoop-namenode-worker247.log
2008-05-15 12:51:39,041 INFO org.apache.hadoop.dfs.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = worker247/192.168.4.247
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.16.0
STARTUP_MSG: build =
http://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.16 -r
618351; compiled by
'hadoopqa' on Mon Feb 4 19:29:11 UTC 2008
************************************************************/
2008-05-15 12:51:39,441 ERROR org.apache.hadoop.dfs.NameNode:
java.net.BindException: Problem binding to maste
r01/192.168.4.1:8020
at org.apache.hadoop.ipc.Server.bind(Server.java:193)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:252)
at org.apache.hadoop.ipc.Server.<init>(Server.java:973)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:393)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:355)
at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:120)
at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:175)
at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:161)
at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:843)
at org.apache.hadoop.dfs.NameNode.main(NameNode.java:852)
2008-05-15 12:51:39,443 INFO org.apache.hadoop.dfs.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at worker247/192.168.4.247
************************************************************/
[3] hadoop-datanode-worker236.log
2008-05-19 12:06:20,839 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 2 for block blk_8110468615207654967 terminating
2008-05-19 12:06:20,839 INFO org.apache.hadoop.dfs.DataNode: writeBlock
blk_8110468615207654967 received exception java.io.EOFException
2008-05-19 12:06:20,839 ERROR org.apache.hadoop.dfs.DataNode:
192.168.4.236:50010:DataXceiver: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at
org.apache.hadoop.dfs.DataNode$BlockReceiver.receiveBlock(DataNode.java:2245)
at
org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1150)
at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:938)
at java.lang.Thread.run(Thread.java:619)
2008-05-19 12:06:26,830 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder blk_8187032148917499050 2 Exception
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
at org.apache.hadoop.dfs.DataNode$PacketResponder.run(DataNode.java:1818)
at java.lang.Thread.run(Thread.java:619)
2008-05-19 12:06:26,830 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 2 for block blk_8187032148917499050 terminating
2008-05-19 12:06:26,830 INFO org.apache.hadoop.dfs.DataNode: writeBlock
blk_8187032148917499050 received exception java.io.EOFException
2008-05-19 12:06:26,830 ERROR org.apache.hadoop.dfs.DataNode:
192.168.4.236:50010:DataXceiver: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at
org.apache.hadoop.dfs.DataNode$BlockReceiver.receiveBlock(DataNode.java:2245)
at
org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1150)
at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:938)
at java.lang.Thread.run(Thread.java:619)
2008-05-19 12:06:29,694 INFO org.apache.hadoop.dfs.DataNode: Deleting
block blk_-363383493122526779 file
/data0/dfs/data/current/subdir54/subdir4/blk_-363383493122526779
2008-05-19 12:06:29,694 INFO org.apache.hadoop.dfs.DataNode: Deleting
block blk_3753472703880843699 file
/data1/dfs/data/current/subdir14/subdir14/blk_3753472703880843699
2008-05-19 12:06:35,982 INFO org.apache.hadoop.dfs.DataBlockScanner:
Verification succeeded for blk_2968960775242567685
2008-05-19 12:06:37,155 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_3449909418507738687 src: /192.168.4.232:55555 dest:
/192.168.4.232:50010
2008-05-19 12:06:37,170 INFO org.apache.hadoop.dfs.DataNode: Datanode 1
got response for connect ack from downstream datanode with firstbadlink as
2008-05-19 12:06:37,171 INFO org.apache.hadoop.dfs.DataNode: Datanode 1
forwarding connect ack to upstream firstbadlink is
2008-05-19 12:06:42,307 INFO org.apache.hadoop.dfs.DataNode: Received
block blk_3449909418507738687 of size 1434183 from /192.168.4.232
2008-05-19 12:06:42,308 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 1 for block blk_3449909418507738687 terminating
2008-05-19 12:06:42,437 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_1416421490768473241 src: /192.168.4.239:43331 dest:
/192.168.4.239:50010
2008-05-19 12:06:42,526 INFO org.apache.hadoop.dfs.DataNode: Datanode 1
got response for connect ack from downstream datanode with firstbadlink as
nstream datanode with firstbadlink as
2008-05-19 12:06:42,526 INFO org.apache.hadoop.dfs.DataNode: Datanode 1
forwarding connect ack to upstream fir
stbadlink is
2008-05-19 12:06:44,301 INFO org.apache.hadoop.dfs.DataNode: Exception
in receiveBlock for block blk_-81297934
64804321952 java.io.EOFException
2008-05-19 12:06:44,672 INFO org.apache.hadoop.dfs.DataNode: Received
block blk_1416421490768473241 of size 14
51550 from /192.168.4.239
2008-05-19 12:06:44,673 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 1 for block blk_14164214907684732
41 terminating
2008-05-19 12:06:45,623 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_-5996804017382201084 src: /19
2.168.4.240:47060 dest: /192.168.4.240:50010
2008-05-19 12:06:45,625 INFO org.apache.hadoop.dfs.DataNode: Datanode 1
got response for connect ack from dow
nstream datanode with firstbadlink as
2008-05-19 12:06:45,625 INFO org.apache.hadoop.dfs.DataNode: Datanode 1
forwarding connect ack to upstream fir
stbadlink is
2008-05-19 12:06:49,999 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_7050204483096558609 src: /192
.168.4.235:40840 dest: /192.168.4.235:50010
2008-05-19 12:06:50,000 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
forwarding connect ack to upstream fir
stbadlink is
2008-05-19 12:06:50,183 INFO org.apache.hadoop.dfs.DataNode: Received
block blk_7050204483096558609 of size 14
02623 from /192.168.4.235
2008-05-19 12:06:50,183 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 0 for block blk_70502044830965586
09 terminating
2008-05-19 12:06:54,435 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_6569064524742381107 src: /192
.168.4.239:43364 dest: /192.168.4.239:50010
2008-05-19 12:06:54,435 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
forwarding connect ack to upstream fir
stbadlink is
2008-05-19 12:06:54,772 INFO org.apache.hadoop.dfs.DataNode: Received
block blk_6569064524742381107 of size 81
23 from /192.168.4.239
2008-05-19 12:06:54,772 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 0 for block blk_65690645247423811
07 terminating
2008-05-19 12:06:55,719 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_4407320173680486712 src: /192
.168.4.228:43545 dest: /192.168.4.228:50010
2008-05-19 12:06:55,722 INFO org.apache.hadoop.dfs.DataNode: Datanode 1
got response for connect ack from dow
nstream datanode with firstbadlink as
2008-05-19 12:06:55,722 INFO org.apache.hadoop.dfs.DataNode: Datanode 1
forwarding connect ack to upstream fir
stbadlink is
2008-05-19 12:06:55,936 INFO org.apache.hadoop.dfs.DataNode: Received
block blk_4407320173680486712 of size 28
74 from /192.168.4.228
2008-05-19 12:06:55,936 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 1 for block blk_44073201736804867
12 terminating
2008-05-19 12:06:56,063 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_-2240603543093865769 src: /19
2.168.4.244:53360 dest: /192.168.4.244:50010
2008-05-19 12:06:56,063 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
forwarding connect ack to upstream fir
stbadlink is
2008-05-19 12:06:56,222 INFO org.apache.hadoop.dfs.DataNode: Received
block blk_-2240603543093865769 of size 1
3330 from /192.168.4.244
2008-05-19 12:06:56,222 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 0 for block blk_-2240603543093865
769 terminating
2008-05-19 12:06:57,179 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at worker236/192.168.4.236
************************************************************/
[4] hadoop-namenode-worker236.log
2008-03-26 11:17:46,538 INFO org.apache.hadoop.dfs.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = worker236/192.168.4.236
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.16.0
STARTUP_MSG: build =
http://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.16 -r
618351; compiled by
'hadoopqa' on Mon Feb 4 19:29:11 UTC 2008
************************************************************/
2008-03-26 11:17:46,632 ERROR org.apache.hadoop.dfs.NameNode:
java.net.BindException: Problem binding to master01/192.168.4.1:8020
at org.apache.hadoop.ipc.Server.bind(Server.java:193)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:252)
at org.apache.hadoop.ipc.Server.<init>(Server.java:973)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:393)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:355)
at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:120)
at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:175)
at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:161)
at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:843)
at org.apache.hadoop.dfs.NameNode.main(NameNode.java:852)
2008-03-26 11:17:46,633 INFO org.apache.hadoop.dfs.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at worker236/192.168.4.236
************************************************************/
Re: Node stops working
Posted by Arun C Murthy <ar...@yahoo-inc.com>.
On May 20, 2008, at 2:21 AM, Marianne Spiller wrote:
> Can anyone please give me a hint?
>
The below "BindException's tell me that there is something already
running on those ports... maybe you need to kill them if they are
Hadoop daemons.
Arun
> 2008-05-15 12:51:55,077 FATAL org.apache.hadoop.mapred.JobTracker:
> java.net.BindException: Problem binding to
> master01/192.168.4.1:9001
> at org.apache.hadoop.ipc.Server.bind(Server.java:193)
> at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:252)
> at org.apache.hadoop.ipc.Server.<init>(Server.java:973)
> at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:393)
> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:355)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:639)
> at org.apache.hadoop.mapred.JobTracker.startTracker
> (JobTracker.java:124)
> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:2114)
> 2008-05-15 12:51:39,441 ERROR org.apache.hadoop.dfs.NameNode:
> java.net.BindException: Problem binding to maste
> r01/192.168.4.1:8020
> at org.apache.hadoop.ipc.Server.bind(Server.java:193)
> at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:252)
> at org.apache.hadoop.ipc.Server.<init>(Server.java:973)
> at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:393)
> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:355)
> at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:120)
> at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:175)
> at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:161)
> at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:843)
> at org.apache.hadoop.dfs.NameNode.main(NameNode.java:852)
>
> 2008-05-15 12:51:39,443 INFO org.apache.hadoop.dfs.NameNode:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at worker247/192.168.4.247
> ************************************************************/
>
>
> [3] hadoop-datanode-worker236.log
> 2008-05-19 12:06:20,839 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 2 for block blk_8110468615207654967 terminating
> 2008-05-19 12:06:20,839 INFO org.apache.hadoop.dfs.DataNode:
> writeBlock blk_8110468615207654967 received exception
> java.io.EOFException
> 2008-05-19 12:06:20,839 ERROR org.apache.hadoop.dfs.DataNode:
> 192.168.4.236:50010:DataXceiver: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.dfs.DataNode$BlockReceiver.receiveBlock
> (DataNode.java:2245)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock
> (DataNode.java:1150)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:938)
> at java.lang.Thread.run(Thread.java:619)
>
> 2008-05-19 12:06:26,830 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder blk_8187032148917499050 2 Exception
> java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:
> 92)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
> at org.apache.hadoop.dfs.DataNode$PacketResponder.run
> (DataNode.java:1818)
> at java.lang.Thread.run(Thread.java:619)
>
> 2008-05-19 12:06:26,830 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 2 for block blk_8187032148917499050 terminating
> 2008-05-19 12:06:26,830 INFO org.apache.hadoop.dfs.DataNode:
> writeBlock blk_8187032148917499050 received exception
> java.io.EOFException
> 2008-05-19 12:06:26,830 ERROR org.apache.hadoop.dfs.DataNode:
> 192.168.4.236:50010:DataXceiver: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.dfs.DataNode$BlockReceiver.receiveBlock
> (DataNode.java:2245)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock
> (DataNode.java:1150)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:938)
> at java.lang.Thread.run(Thread.java:619)
>
> 2008-05-19 12:06:29,694 INFO org.apache.hadoop.dfs.DataNode:
> Deleting block blk_-363383493122526779 file /data0/dfs/data/current/
> subdir54/subdir4/blk_-363383493122526779
> 2008-05-19 12:06:29,694 INFO org.apache.hadoop.dfs.DataNode:
> Deleting block blk_3753472703880843699 file /data1/dfs/data/current/
> subdir14/subdir14/blk_3753472703880843699
> 2008-05-19 12:06:35,982 INFO
> org.apache.hadoop.dfs.DataBlockScanner: Verification succeeded for
> blk_2968960775242567685
> 2008-05-19 12:06:37,155 INFO org.apache.hadoop.dfs.DataNode:
> Receiving block blk_3449909418507738687 src: /192.168.4.232:55555
> dest: /192.168.4.232:50010
> 2008-05-19 12:06:37,170 INFO org.apache.hadoop.dfs.DataNode:
> Datanode 1 got response for connect ack from downstream datanode
> with firstbadlink as
> 2008-05-19 12:06:37,171 INFO org.apache.hadoop.dfs.DataNode:
> Datanode 1 forwarding connect ack to upstream firstbadlink is
> 2008-05-19 12:06:42,307 INFO org.apache.hadoop.dfs.DataNode:
> Received block blk_3449909418507738687 of size 1434183 from /
> 192.168.4.232
> 2008-05-19 12:06:42,308 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 1 for block blk_3449909418507738687 terminating
> 2008-05-19 12:06:42,437 INFO org.apache.hadoop.dfs.DataNode:
> Receiving block blk_1416421490768473241 src: /192.168.4.239:43331
> dest: /192.168.4.239:50010
> 2008-05-19 12:06:42,526 INFO org.apache.hadoop.dfs.DataNode:
> Datanode 1 got response for connect ack from downstream datanode
> with firstbadlink as
> nstream datanode with firstbadlink as
> 2008-05-19 12:06:42,526 INFO org.apache.hadoop.dfs.DataNode:
> Datanode 1 forwarding connect ack to upstream fir
> stbadlink is
> 2008-05-19 12:06:44,301 INFO org.apache.hadoop.dfs.DataNode:
> Exception in receiveBlock for block blk_-81297934
> 64804321952 java.io.EOFException
> 2008-05-19 12:06:44,672 INFO org.apache.hadoop.dfs.DataNode:
> Received block blk_1416421490768473241 of size 14
> 51550 from /192.168.4.239
> 2008-05-19 12:06:44,673 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 1 for block blk_14164214907684732
> 41 terminating
> 2008-05-19 12:06:45,623 INFO org.apache.hadoop.dfs.DataNode:
> Receiving block blk_-5996804017382201084 src: /19
> 2.168.4.240:47060 dest: /192.168.4.240:50010
> 2008-05-19 12:06:45,625 INFO org.apache.hadoop.dfs.DataNode:
> Datanode 1 got response for connect ack from dow
> nstream datanode with firstbadlink as
> 2008-05-19 12:06:45,625 INFO org.apache.hadoop.dfs.DataNode:
> Datanode 1 forwarding connect ack to upstream fir
> stbadlink is
> 2008-05-19 12:06:49,999 INFO org.apache.hadoop.dfs.DataNode:
> Receiving block blk_7050204483096558609 src: /192
> .168.4.235:40840 dest: /192.168.4.235:50010
> 2008-05-19 12:06:50,000 INFO org.apache.hadoop.dfs.DataNode:
> Datanode 0 forwarding connect ack to upstream fir
> stbadlink is
> 2008-05-19 12:06:50,183 INFO org.apache.hadoop.dfs.DataNode:
> Received block blk_7050204483096558609 of size 14
> 02623 from /192.168.4.235
> 2008-05-19 12:06:50,183 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_70502044830965586
> 09 terminating
> 2008-05-19 12:06:54,435 INFO org.apache.hadoop.dfs.DataNode:
> Receiving block blk_6569064524742381107 src: /192
> .168.4.239:43364 dest: /192.168.4.239:50010
> 2008-05-19 12:06:54,435 INFO org.apache.hadoop.dfs.DataNode:
> Datanode 0 forwarding connect ack to upstream fir
> stbadlink is
> 2008-05-19 12:06:54,772 INFO org.apache.hadoop.dfs.DataNode:
> Received block blk_6569064524742381107 of size 81
> 23 from /192.168.4.239
> 2008-05-19 12:06:54,772 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_65690645247423811
> 07 terminating
> 2008-05-19 12:06:55,719 INFO org.apache.hadoop.dfs.DataNode:
> Receiving block blk_4407320173680486712 src: /192
> .168.4.228:43545 dest: /192.168.4.228:50010
> 2008-05-19 12:06:55,722 INFO org.apache.hadoop.dfs.DataNode:
> Datanode 1 got response for connect ack from dow
> nstream datanode with firstbadlink as
> 2008-05-19 12:06:55,722 INFO org.apache.hadoop.dfs.DataNode:
> Datanode 1 forwarding connect ack to upstream fir
> stbadlink is
> 2008-05-19 12:06:55,936 INFO org.apache.hadoop.dfs.DataNode:
> Received block blk_4407320173680486712 of size 28
> 74 from /192.168.4.228
> 2008-05-19 12:06:55,936 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 1 for block blk_44073201736804867
> 12 terminating
> 2008-05-19 12:06:56,063 INFO org.apache.hadoop.dfs.DataNode:
> Receiving block blk_-2240603543093865769 src: /19
> 2.168.4.244:53360 dest: /192.168.4.244:50010
> 2008-05-19 12:06:56,063 INFO org.apache.hadoop.dfs.DataNode:
> Datanode 0 forwarding connect ack to upstream fir
> stbadlink is
> 2008-05-19 12:06:56,222 INFO org.apache.hadoop.dfs.DataNode:
> Received block blk_-2240603543093865769 of size 1
> 3330 from /192.168.4.244
> 2008-05-19 12:06:56,222 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_-2240603543093865
> 769 terminating
> 2008-05-19 12:06:57,179 INFO org.apache.hadoop.dfs.DataNode:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at worker236/192.168.4.236
> ************************************************************/
>
>
> [4] hadoop-namenode-worker236.log
> 2008-03-26 11:17:46,538 INFO org.apache.hadoop.dfs.NameNode:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = worker236/192.168.4.236
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.16.0
> STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/core/
> branches/branch-0.16 -r 618351; compiled by
> 'hadoopqa' on Mon Feb 4 19:29:11 UTC 2008
> ************************************************************/
> 2008-03-26 11:17:46,632 ERROR org.apache.hadoop.dfs.NameNode:
> java.net.BindException: Problem binding to master01/192.168.4.1:8020
> at org.apache.hadoop.ipc.Server.bind(Server.java:193)
> at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:252)
> at org.apache.hadoop.ipc.Server.<init>(Server.java:973)
> at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:393)
> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:355)
> at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:120)
> at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:175)
> at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:161)
> at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:843)
> at org.apache.hadoop.dfs.NameNode.main(NameNode.java:852)
>
> 2008-03-26 11:17:46,633 INFO org.apache.hadoop.dfs.NameNode:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at worker236/192.168.4.236
> ************************************************************/