You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Viliam Holub <vi...@ucd.ie> on 2011/05/09 04:02:17 UTC

Shutting down HDFS client

Does anybody know what might be the root cause of this exception:

11/05/08 17:07:28 WARN hdfs.DFSClient: Problem renewing lease for DFSClient_1994238103 for a period of 0 seconds. Shutting down HDFS client...
java.net.SocketTimeoutException: Call to dfs-master/10.235.79.159:8020 failed on socket timeout exception: java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=dfs-master/10.235.79.159:8020]
        at org.apache.hadoop.ipc.Client.wrapException(Client.java:936)
        at org.apache.hadoop.ipc.Client.call(Client.java:908)
        at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
        at $Proxy5.renewLease(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at $Proxy5.renewLease(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1366)
        at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1382)
        at java.lang.Thread.run(Thread.java:662)

I found this in datanode's log if it is related:

2011-05-08 17:07:28,686 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block blk_-438222431049744157_3318 java.io.EOFException: while trying to read 51692 bytes
2011-05-08 17:07:28,687 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder blk_-438222431049744157_3318 2 : Thread is interrupted.
2011-05-08 17:07:28,687 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 2 for block blk_-438222431049744157_3318 terminating
2011-05-08 17:07:28,687 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-438222431049744157_3318 received exception java.io.EOFException: while trying to read 51692 bytes
2011-05-08 17:07:28,687 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.235.79.159:50010, storageID=DS-1501276445-10.235.79.159-50010-1303773028829, infoPort=50075, ipcPort=50020):DataXceiver
java.io.EOFException: while trying to read 51692 bytes
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:309)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:353)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:409)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:617)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.opWriteBlock(DataXceiver.java:352)
        at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.opWriteBlock(DataTransferProtocol.java:390)
        at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.processOp(DataTransferProtocol.java:331)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:111)
        at java.lang.Thread.run(Thread.java:662)

This pattern in repeated several times within few seconds.

Thanks a lot,
Viliam