You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Glen Geng (Jira)" <ji...@apache.org> on 2021/03/26 05:40:00 UTC
[jira] [Commented] (HDDS-5033) SCM may not be able to know full
port list of Datanode after Datanode is restarted.
[ https://issues.apache.org/jira/browse/HDDS-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309178#comment-17309178 ]
Glen Geng commented on HDDS-5033:
---------------------------------
[~adoroszlai] Could you please confirm this issue ?
> SCM may not be able to know full port list of Datanode after Datanode is restarted.
> -----------------------------------------------------------------------------------
>
> Key: HDDS-5033
> URL: https://issues.apache.org/jira/browse/HDDS-5033
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Affects Versions: 1.2.0
> Reporter: Glen Geng
> Priority: Major
> Attachments: 企业微信截图_097abd79-0ea4-487b-9b07-6bc2330385ef.png, 企业微信截图_c0bd5dde-98ee-4350-914d-2e0069ea8602.png
>
>
> Please check attachment.
>
> After restart DN, the SCM may not know the full ports of that DN.
>
> This issue can not be solved without restart SCM. The consequence is that Datanode can not participate any pipeline, and there will be continually NPE in DN.
> {code:java}
> 2021-03-25 15:04:16,322 [Command processor thread] ERROR org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine: Critical Error : Command processor thread encountered an error. Thread: Thread[Command processor thread,5,main]
> java.lang.NullPointerException
> at org.apache.hadoop.hdds.ratis.RatisHelper.toRaftPeerAddress(RatisHelper.java:99)
> at org.apache.hadoop.hdds.ratis.RatisHelper.raftPeerBuilderFor(RatisHelper.java:119)
> at org.apache.hadoop.hdds.ratis.RatisHelper.toRaftPeer(RatisHelper.java:111)
> at org.apache.hadoop.hdds.ratis.RatisHelper.newRaftGroup(RatisHelper.java:149)
> at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CreatePipelineCommandHandler.handle(CreatePipelineCommandHandler.java:91)
> at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:99)
> at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$2(DatanodeStateMachine.java:506)
> at java.lang.Thread.run(Thread.java:748)
> 2021-03-25 15:04:16,323 [Command processor thread] ERROR org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine: Critical Error : Command processor thread encountered an error. Thread: Thread[Command processor thread,5,main]
> java.lang.NullPointerException
> at org.apache.hadoop.hdds.ratis.RatisHelper.toRaftPeerAddress(RatisHelper.java:99)
> at org.apache.hadoop.hdds.ratis.RatisHelper.raftPeerBuilderFor(RatisHelper.java:119)
> at org.apache.hadoop.hdds.ratis.RatisHelper.toRaftPeer(RatisHelper.java:111)
> at org.apache.hadoop.hdds.ratis.RatisHelper.newRaftGroup(RatisHelper.java:149)
> at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CreatePipelineCommandHandler.handle(CreatePipelineCommandHandler.java:91)
> at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:99)
> at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$2(DatanodeStateMachine.java:506)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> After restart SCM, this issue gone.
> The issue should be: SCMNodeManager just record the DatanodeDetails once during register.
> But for DN, it won’t record the admin, server, client port into DatanodeDetails until its ratis server is up.
> Thus there is contention here: if the register request is reported before ratis server is up, SCM won’t know full port list of that DN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org