You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "刘珍 (Jira)" <ji...@apache.org> on 2022/11/04 06:47:00 UTC
[jira] [Created] (IOTDB-4851) 1rep 3C 5D , remove 1 datanode , region migration failed（Unknown）
刘珍 created IOTDB-4851:
-------------------------
Summary: 1rep 3C 5D , remove 1 datanode , region migration failed（Unknown）
Key: IOTDB-4851
URL: https://issues.apache.org/jira/browse/IOTDB-4851
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Gaofei Cao
Attachments: image-2022-11-04-14-40-54-764.png
版本：m_1103_f857667
{color:#DE350B}1副本{color}，3C5D
schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.{color:#DE350B}MultiLeaderConsensus{color}
缩容1节点（ip76），从集群中移除此节点成功，但是：
问题1 ： 缩容节点的data没有迁移成功
!image-2022-11-04-14-40-54-764.png!
问题2：缩容节点的datanode进程不退出，datanode日志刷：
2022-11-04 13:52:56,659 [7@group-000200000006-SegmentedRaftLogWorker] INFO o.a.r.s.r.s.SegmentedRaftLogWorker:345 - 7@group-000200000006-SegmentedRaftLogWorker was interrupted, exiting. There are 0 tasks remaining in the queue.
2022-11-04 13:52:56,659 [7@group-000200000001-SegmentedRaftLogWorker] INFO o.a.r.s.r.s.SegmentedRaftLogWorker:345 - 7@group-000200000001-SegmentedRaftLogWorker was interrupted, exiting. There are 0 tasks remaining in the queue.
2022-11-04 13:52:56,684 [7-impl-thread2] INFO o.a.r.s.r.s.SegmentedRaftLogWorker:255 - 7@group-000200000006-SegmentedRaftLogWorker close()
2022-11-04 13:52:56,701 [7-impl-thread3] INFO o.a.r.s.r.s.SegmentedRaftLogWorker:255 - 7@group-000200000001-SegmentedRaftLogWorker close()
2022-11-04 13:52:56,703 [JvmPauseMonitor0] INFO o.a.r.u.JvmPauseMonitor:111 - JvmPauseMonitor-7: Stopped
2022-11-04 13:52:56,705 [pool-24-IoTDB-DataNodeInternalRPC-Processor-146] INFO o.a.i.c.s.ThriftService:158 - IoTDB: closing Multi Leader consensus Service...
2022-11-04 13:52:56,707 [pool-24-IoTDB-DataNodeInternalRPC-Processor-146] INFO o.a.i.c.s.ThriftService:165 - IoTDB: close Multi Leader consensus Service successfully
2022-11-04 13:52:56,707 [pool-24-IoTDB-DataNodeInternalRPC-Processor-146] INFO o.a.i.c.s.RegisterManager:67 - deregister all service.
2022-11-04 13:53:23,771 [DataNodeInternalRPC-Service] ERROR o.a.t.s.TThreadPoolServer:144 - Shutdown is not done after 60SECONDS
2022-11-04 13:53:23,778 [MPPDataExchangeRPC-Service] ERROR o.a.t.s.TThreadPoolServer:144 - Shutdown is not done after 60SECONDS
2022-11-04 13:53:54,354 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 60s to wait for closing all TsFiles.
2022-11-04 13:54:54,354 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 120s to wait for closing all TsFiles.
2022-11-04 13:55:54,354 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 180s to wait for closing all TsFiles.
2022-11-04 13:56:54,355 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 240s to wait for closing all TsFiles.
2022-11-04 13:57:54,355 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 300s to wait for closing all TsFiles.
2022-11-04 13:58:54,355 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 360s to wait for closing all TsFiles.
2022-11-04 13:59:54,356 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 420s to wait for closing all TsFiles.
2022-11-04 14:00:54,356 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 480s to wait for closing all TsFiles.
2022-11-04 14:01:54,357 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 540s to wait for closing all TsFiles.
2022-11-04 14:02:54,357 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 600s to wait for closing all TsFiles.
2022-11-04 14:03:54,357 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 660s to wait for closing all TsFiles.
2022-11-04 14:04:54,358 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 720s to wait for closing all TsFiles.
2022-11-04 14:05:54,358 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 780s to wait for closing all TsFiles.
2022-11-04 14:06:54,358 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 840s to wait for closing all TsFiles.
2022-11-04 14:07:54,358 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 900s to wait for closing all TsFiles.
2022-11-04 14:08:54,359 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 960s to wait for closing all TsFiles.
2022-11-04 14:09:54,359 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 1020s to wait for closing all TsFiles.
2022-11-04 14:10:54,359 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 1080s to wait for closing all TsFiles.
2022-11-04 14:11:54,360 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 1140s to wait for closing all TsFiles.
2022-11-04 14:12:54,360 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 1200s to wait for closing all TsFiles.
2022-11-04 14:13:54,360 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 1260s to wait for closing all TsFiles.
2022-11-04 14:14:54,360 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 1320s to wait for closing all TsFiles.
2022-11-04 14:15:54,361 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 1380s to wait for closing all TsFiles.
2022-11-04 14:16:54,361 [Thread-0] WARN o.a.i.d.e.s.DataRegion:1848 - root.test.g_2-21 has spent 1440s to wait for closing all TsFiles.
……
测试流程
1. 启动集群 192.168.10.72~75
ConfigNode 72，73，74
MAX_HEAP_SIZE="8G"
Common
max_connection_for_internal_service=300
query_timeout_threshold=3600000
schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
schema_replication_factor=1
data_replication_factor=1
DataNode
MAX_HEAP_SIZE="256G"
MAX_DIRECT_MEMORY_SIZE="32G"
2. bm写入数据
配置见附件
3. 写入完成，无其他客户端操作，缩容ip76
详细日志见机器
/data/mpp_test/m_1103_f857667
--
This message was sent by Atlassian Jira
(v8.20.10#820010)