You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "zy.jordan (Jira)" <ji...@apache.org> on 2019/11/19 06:03:00 UTC
[jira] [Commented] (HDFS-7877) [Umbrella] Support maintenance state
for datanodes
[ https://issues.apache.org/jira/browse/HDFS-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977142#comment-16977142 ]
zy.jordan commented on HDFS-7877:
---------------------------------
Hi, the maintenance is a great feature. But I have encounter a little problem.
When maintenance time is expired, and some nodes can't get out of maintenance state. And print some logs.
{quote}2019-11-18 18:45:11,389 INFO [DecommissionMonitor-0] BlockStateChange: BLOCK* InvalidateBlocks: add blk_52447675210_51376569902 to xxxxxxxxip1:50010
2019-11-18 18:45:11,389 INFO [DecommissionMonitor-0] org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated 38463 over-replicated blocks on 10.208.51.61:50010 during recommissioning
2019-11-18 18:45:11,389 WARN [DecommissionMonitor-0] org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager: DecommissionManager caught exception when processing node.
2019-11-18 18:45:11,389 INFO [DecommissionMonitor-0] org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem write lock held for 1812 ms via
org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor.run(DecommissionManager.java:499)
2019-11-18 18:45:11,389 INFO [DecommissionMonitor-0] org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager: Checked 0 blocks and 2 nodes this tick
2019-11-18 18:45:24,577 INFO [DecommissionMonitor-0] org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager: Stopping maintenance of live node 10.208.51.69:50010
2019-11-18 18:45:24,577 INFO [DecommissionMonitor-0] BlockStateChange: BLOCK* InvalidateBlocks: add blk_1164445189_90720573 to xxxxxxxxxip2:50010
{quote}
{quote}2019-11-18 19:02:10,467 WARN [DecommissionMonitor-0] org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager: DatanodeAdminMonitor caught exception when processing node xxxxxxxxxip1:50010.
java.lang.IllegalStateException: Node xxxxxxxxxip1:50010 is in an invalid state: In Service Invalid state: 0 blocks are on this dn
at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor.check(DecommissionManager.java:587)
at org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor.run(DecommissionManager.java:495)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{quote}
It shows when it get one node out of maintenance state, it caught +_*exception when processing node*_+ .
> [Umbrella] Support maintenance state for datanodes
> --------------------------------------------------
>
> Key: HDFS-7877
> URL: https://issues.apache.org/jira/browse/HDFS-7877
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
> Reporter: Ming Ma
> Assignee: Ming Ma
> Priority: Major
> Fix For: 2.9.0, 3.0.0-beta1, 3.1.0
>
> Attachments: HDFS-7877-2.patch, HDFS-7877.patch, Supportmaintenancestatefordatanodes-2.pdf, Supportmaintenancestatefordatanodes.pdf
>
>
> This requirement came up during the design for HDFS-7541. Given this feature is mostly independent of upgrade domain feature, it is better to track it under a separate jira. The design and draft patch will be available soon.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org