You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "刘珍 (Jira)" <ji...@apache.org> on 2023/03/08 03:33:00 UTC
[jira] [Assigned] (IOTDB-4593) [Remove-DataNode] Removing nodes writes data
[ https://issues.apache.org/jira/browse/IOTDB-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
刘珍 reassigned IOTDB-4593:
-------------------------
Component/s: mpp-cluster
Sprint: 2023-1-ConfigNode, 2023-2-Catalyst (was: 2023-1-ConfigNode)
Assignee: Xinyu Tan (was: Gaofei Cao)
> [Remove-DataNode] Removing nodes writes data
> --------------------------------------------
>
> Key: IOTDB-4593
> URL: https://issues.apache.org/jira/browse/IOTDB-4593
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
> Affects Versions: 0.14.0-SNAPSHOT
> Reporter: 刘珍
> Assignee: Xinyu Tan
> Priority: Major
> Attachments: image-2022-10-10-13-36-14-475.png, image-2023-03-08-11-29-52-352.png, image-2023-03-08-11-30-38-559.png, image-2023-03-08-11-30-51-220.png, more_dev.conf, screenshot-1.png
>
>
> m_0930_2a30316
> 问题描述：
> 缩容datanode，{color:#DE350B}*节点置为Removing状态，但是在继续接受写入*{color}（benchmark运行1小时，执行缩容，*耗时3小时*，缩容完成）：
> 2022-10-08 13:23:54,686 [pool-20-IoTDB-DataNodeInternalRPC-Processor-148] INFO o.a.i.c.conf.CommonConfig:305 - *Set system mode from Running to Removing*.
> Removing状态后（create 207个tsfile），
> !image-2022-10-10-13-36-14-475.png!
> 测试环境
> 1. 192.168.10.71-76 6台物理机 48cpu 384GB
> 3C : 192.168.10.72 , 73,74
> 5D : 192.168.10.72 , 73,74 , 75 , 76
> benchmark：192.168.10.71
> ConfigNode配置参数
> MAX_HEAP_SIZE="8G"
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
> schema_replication_factor=3
> data_replication_factor=3
> connection_timeout_ms=120000
> DataNode配置参数
> MAX_HEAP_SIZE="256G"
> MAX_DIRECT_MEMORY_SIZE="32G"
> connection_timeout_ms=120000
> max_connection_for_internal_service=200
> max_waiting_time_when_insert_blocked=600000
> query_timeout_threshold=36000000
> 2. benchmark 配置文件见附件
> GROUP_NUMBER=10
> DEVICE_NUMBER=50000
> SENSOR_NUMBER=600
> IS_OUT_OF_ORDER=false
> OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0
> CLIENT_NUMBER=100
> LOOP=100
> BATCH_SIZE_PER_WRITE=100
> 3. 运行1小时，ip72缩容
> liuzhen@fit-72:/data/mpp_test/m_0930_2a30316/datanode$ cat 1008_test_remove_1h.sh
> sleep 1h
> /data/mpp_test/m_0930_2a30316/datanode/sbin/start-cli.sh -h 192.168.10.72 -e "show cluster" > 1008_3c5d_bef_remove.out
> /data/mpp_test/m_0930_2a30316/datanode/sbin/start-cli.sh -h 192.168.10.72 -e "show regions" >> 1008_3c5d_bef_remove.out
> /data/mpp_test/m_0930_2a30316/datanode/sbin/remove-datanode.sh "192.168.10.72:6667" >> 1008_3c5d_1hour_remove_ip72.out
> 4. ip72 的日志见机器上的备份
> /data/mpp_test/m_0930_2a30316/datanode/logs_bm_1h_remove_ip72
--
This message was sent by Atlassian Jira
(v8.20.10#820010)