You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Ramya R (JIRA)" <ji...@apache.org> on 2009/02/25 12:28:02 UTC

[jira] Commented: (HADOOP-5324) Reduce step hangs while recovering a block from bad datanode

    [ https://issues.apache.org/jira/browse/HADOOP-5324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676613#action_12676613 ] 

Ramya R commented on HADOOP-5324:
---------------------------------

Below is the last 4KB of the log file. With the below message reducer hangs forever.
\\ \\ 
{noformat}
2009-02-25 08:56:36,584 INFO org.apache.hadoop.mapred.ReduceTask: GetMapEventsThread exiting
2009-02-25 08:56:36,584 INFO org.apache.hadoop.mapred.ReduceTask: getMapsEventsThread joined.
2009-02-25 08:56:36,585 INFO org.apache.hadoop.mapred.ReduceTask: Closed ram manager
2009-02-25 08:56:36,585 INFO org.apache.hadoop.mapred.ReduceTask: Interleaved on-disk merge complete: 12 files left.
2009-02-25 08:56:36,586 INFO org.apache.hadoop.mapred.ReduceTask: In-memory merge complete: 15 files left.
2009-02-25 08:56:36,589 INFO org.apache.hadoop.mapred.ReduceTask: Keeping 15 segments, 66668024 bytes in memory for intermediate, on-disk merge
2009-02-25 08:56:36,597 INFO org.apache.hadoop.mapred.ReduceTask: Merging 12 files, 1072060708 bytes from disk
2009-02-25 08:56:36,599 INFO org.apache.hadoop.mapred.ReduceTask: Merging 0 segments, 0 bytes from memory into reduce
2009-02-25 08:56:36,599 INFO org.apache.hadoop.mapred.Merger: Merging 27 sorted segments
2009-02-25 08:56:49,155 INFO org.apache.hadoop.mapred.Merger: Merging 18 intermediate segments out of a total of 27
2009-02-25 08:59:44,318 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 1072060626 bytes
2009-02-25 09:21:26,758 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block <blk_ID1>java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/<hostname1:port1> remote=/<hostname1:port2>]
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.DataInputStream.readFully(DataInputStream.java:178)
	at java.io.DataInputStream.readLong(DataInputStream.java:399)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2333)

2009-02-25 09:21:26,759 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block <blk_ID1> bad datanode[0] <hostname1:port2>
2009-02-25 09:21:26,759 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block <blk_ID1> in pipeline <hostname1:port2>, <hostname2:port1>, <hostname3:port1>: bad datanode <hostname1:port2>
2009-02-25 09:22:38,114 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/<hostname1:port3> remote=/<hostname1:port2>]
2009-02-25 09:22:38,260 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block <blk_ID2>
2009-02-25 09:22:38,263 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target node: <hostname1:port2>
2009-02-25 10:06:35,909 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block <blk_ID3>java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/<hostname1:port4> remote=/<hostname1:port2>]
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.DataInputStream.readFully(DataInputStream.java:178)
	at java.io.DataInputStream.readLong(DataInputStream.java:399)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2333)

2009-02-25 10:06:43,847 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block <blk_ID3> bad datanode[0] <hostname1:port2>
2009-02-25 10:06:43,847 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block <blk_ID3> in pipeline <hostname1:port2>, <hostname4:port1>, <hostname5:port1>: bad datanode <hostname1:port2>
{noformat}

> Reduce step hangs while recovering a block from bad datanode
> ------------------------------------------------------------
>
>                 Key: HADOOP-5324
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5324
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Ramya R
>             Fix For: 0.20.0
>
>
> The reduce step hangs infinitely when its trying to recover a block from a bad datanode. The node from which the block is being retrieved is alive and TT and DN are up and running.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.