You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2010/12/28 19:40:48 UTC
[jira] Issue Comment Edited: (HDFS-1540) Make Datanode handle
errors to namenode.register call more elegantly
[ https://issues.apache.org/jira/browse/HDFS-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12975577#action_12975577 ]
dhruba borthakur edited comment on HDFS-1540 at 12/28/10 1:39 PM:
------------------------------------------------------------------
Patch also uploaded at https://reviews.apache.org/r/195/
Thanks Todd for the review. I meant to do this for the register call, but my merge from our 0.20 based tree into apache trunk caused the earlier error. For background: when we restarted one of our namenodes, many datanodes exited out with the following stack trace:
{code}
Retrying connect to server: zzz.com/yyy:9005. Already tried 5 time(s).
org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to yyy.com/:9005 failed on local exception: java.io.IOException: Connection reset by peer
at org.apache.hadoop.ipc.Client.wrapException(Client.java:796)
at org.apache.hadoop.ipc.Client.call(Client.java:764)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.register(Unknown Source)
at org.apache.hadoop.hdfs.server.datanode.DataNode.register(DataNode.java:554)
at org.apache.hadoop.hdfs.server.datanode.DataNode.runDatanodeDaemon(DataNode.java:1281)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1320)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1441)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:206)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
{code}
This caused the restart to get slowed down because many blocks were missing and the NN refused to exit safemode automatically as expected.
The fix here is to avoid the datanode from exiting itself unless a explicit shutdown-exception is thrown by the NN. Otherwise, the datanode should never exit and should continuously try to establish contact with the NN.
was (Author: dhruba):
Patch also uploaded at https://reviews.apache.org/r/195/
Thanks Todd for the review. I meant to do this for the register call, but my merge from our 0.20 based tree into apache trunk caused the earlier error. For background: when we restarted one of our namenodes, many datanodes exited out with the following stack trace:
{code}
Retrying connect to server: dfstmp.data.facebook.com/10.26.108.201:9005. Already tried 5 time(s).
org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to yyy.com/:9005 failed on local exception: java.io.IOException: Connection reset by peer
at org.apache.hadoop.ipc.Client.wrapException(Client.java:796)
at org.apache.hadoop.ipc.Client.call(Client.java:764)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.register(Unknown Source)
at org.apache.hadoop.hdfs.server.datanode.DataNode.register(DataNode.java:554)
at org.apache.hadoop.hdfs.server.datanode.DataNode.runDatanodeDaemon(DataNode.java:1281)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1320)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1441)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:206)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
{code}
This caused the restart to get slowed down because many blocks were missing and the NN refused to exit safemode automatically as expected.
The fix here is to avoid the datanode from exiting itself unless a explicit shutdown-exception is thrown by the NN. Otherwise, the datanode should never exit and should continuously try to establish contact with the NN.
> Make Datanode handle errors to namenode.register call more elegantly
> --------------------------------------------------------------------
>
> Key: HDFS-1540
> URL: https://issues.apache.org/jira/browse/HDFS-1540
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: datanodeException1.txt, datanodeException2.txt
>
>
> When a datanode receives a "Connection reset by peer" from the namenode.register(), it exits. This causes many datanodes to die.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.