You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2025/12/19 00:26:00 UTC
[jira] [Commented] (HDFS-14857) FS operations fail in HA mode: DataNode fails to connect to NameNode
[ https://issues.apache.org/jira/browse/HDFS-14857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18046468#comment-18046468 ]
ASF GitHub Bot commented on HDFS-14857:
---------------------------------------
github-actions[bot] commented on PR #1480:
URL: https://github.com/apache/hadoop/pull/1480#issuecomment-3672853681
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.
> FS operations fail in HA mode: DataNode fails to connect to NameNode
> --------------------------------------------------------------------
>
> Key: HDFS-14857
> URL: https://issues.apache.org/jira/browse/HDFS-14857
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 3.1.0
> Reporter: Jeff Saremi
> Assignee: Jeff Saremi
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In an HA configuration, if the NameNodes get restarted and if they're assigned new IP addresses, any client FS operation such as a copyFromLocal will fail with a message like the following:
> {{2019-09-12 18:47:30,544 WARN hdfs.DataStreamer: DataStreamer Exceptionorg.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/init.sh._COPYING_ could only be written to 0 of the 1 minReplication nodes. There are 2 datanode(s) running and 2 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2211) ...}}
>
> Looking at DataNode's stderr shows the following:
> * The heartbeat service detects the IP change and recovers (almost)
> * At this stage, an *hdfs dfsadmin -report* reports all datanodes correctly
> * Once the write begins, the follwoing exception shows up in the datanode log: *no route to host*
> {{2019-09-12 01:35:11,251 WARN datanode.DataNode: IOException in offerService2019-09-12 01:35:11,251 WARN datanode.DataNode: IOException in offerServicejava.io.EOFException: End of File Exception between local host is: "storage-0-0.storage-0-svc.test.svc.cluster.local/10.244.0.211"; destination host is: "nmnode-0-0.nmnode-0-svc.test.svc.cluster.local":9000; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:789) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549) at org.apache.hadoop.ipc.Client.call(Client.java:1491) at org.apache.hadoop.ipc.Client.call(Client.java:1388) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy17.sendHeartbeat(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:166) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:516) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:646) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:847) at java.lang.Thread.run(Thread.java:748)Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1850) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1183) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079)}}
> {{2019-09-12 01:41:12,273 WARN ipc.Client: Address change detected. Old: nmnode-0-1.nmnode-0-svc.test.svc.cluster.local/10.244.0.217:9000 New: nmnode-0-1.nmnode-0-svc.test.svc.cluster.local/10.244.0.220:9000}}{{...}}
>
> {{2019-09-12 01:41:12,482 INFO datanode.DataNode: Block pool BP-930210564-10.244.0.216-1568249865477 (Datanode Uuid 7673ef28-957a-439f-a721-d47a4a6adb7b) service to nmnode-0-1.nmnode-0-svc.test.svc.cluster.local/10.244.0.217:9000 beginning handshake with NN}}
> {{2019-09-12 01:41:12,534 INFO datanode.DataNode: Block pool Block pool BP-930210564-10.244.0.216-1568249865477 (Datanode Uuid 7673ef28-957a-439f-a721-d47a4a6adb7b) service to nmnode-0-1.nmnode-0-svc.test.svc.cluster.local/10.244.0.217:9000 successfully registered with NN}}
>
> *NOTE*: See how when the '{{Address change detected' shows up, the printout correctly shows the old and the new address (}}{{10.244.0.220}}{{). }}{{However when the registration with NN is complete, the old IP address is still being printed (}}{{10.244.0.217) which is showing how cached copies of the IP addresses linger on.}}{{}}
>
> {{And the following is where the actual error happens preventing any writes to FS:}}
>
> {{2019-09-12 18:45:29,843 INFO retry.RetryInvocationHandler: java.net.NoRouteToHostException: No Route to Host from storage-0-0.storage-0-svc.test.svc.cluster.local/10.244.0.211 to nmnode-0-1.nmnode-0-svc:50200 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see: http://wiki.apache.org/hadoop/NoRouteToHost, while invoking InMemoryAliasMapProtocolClientSideTranslatorPB.read over nmnode-0-1.nmnode-0-svc/10.244.0.217:50200 after 3 failover attempts. Trying to failover after sleeping for 4452ms.}}{{}}{{}}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org