[jira] [Commented] (IOTDB-3802) [ raft log ] Raft log takes up a lot of disk space

["刘珍 (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IOTDB-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17624808#comment-17624808 ] 

刘珍 commented on IOTDB-3802:
---------------------------

master_1025_fb65687
192.168.10.68/62/66
ConfigNode
MAX_HEAP_SIZE="8G"
schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
data_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
schema_replication_factor=3
data_replication_factor=3
DataNode
MAX_HEAP_SIZE="192G"
MAX_DIRECT_MEMORY_SIZE="32G"
consensus_dir=/data/liuzhen_test/master_1025_fb65687/data_consensus
benchmark见附件（在ip64）
如下：{color:#DE350B}*raft log 有1.5TB，太大，而且没有做snapshot*{color}。
 !screenshot-2.png! 

BENCHMARK主要配置
GROUP_NUMBER=10
DEVICE_NUMBER=50000
REAL_INSERT_RATE=1.0
SENSOR_NUMBER=600
IS_SENSOR_TS_ALIGNMENT=true
IS_OUT_OF_ORDER=false
OUT_OF_ORDER_RATIO=0.5
########### Data Amount ###########
OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0
CLIENT_NUMBER=100
LOOP=100
BATCH_SIZE_PER_WRITE=100
START_TIME=2019-8-30T00:00:00+08:00
POINT_STEP=1000
OP_INTERVAL=0
INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1
ENCODINGS=PLAIN/PLAIN/PLAIN/PLAIN/PLAIN/PLAIN
COMPRESSOR=SNAPPY
########### Other Param ###########
IS_DELETE_DATA=false
CREATE_SCHEMA=true


> [ raft log ] Raft log takes up a lot of disk space
> --------------------------------------------------
>
>                 Key: IOTDB-3802
>                 URL: https://issues.apache.org/jira/browse/IOTDB-3802
>             Project: Apache IoTDB
>          Issue Type: Improvement
>          Components: mpp-cluster
>            Reporter: 刘珍
>            Assignee: Song Ziyang
>            Priority: Major
>         Attachments: config.properties, image-2022-07-12-10-41-14-126.png, screenshot-1.png, screenshot-2.png
>
>
> master_0711_d1dd425 
> RatisConsensus ,3副本3C3D
> benchmark写入完成（配置文件见附件），所有节点同步完成，执行flush。
> raft log占用磁盘空间大：
> 4个dataregion的raft log分别是8.3GB/5.8GB/46GB/27GB
> 如果节点数/dataregion数比较多，磁盘占用会更大。需优化。
>  !image-2022-07-12-10-41-14-126.png! 
> 复现流程
> 1. 192.168.10.62 /66 /68
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> schema_replication_factor=3
> data_replication_factor=3
> max_waiting_time_when_insert_blocked=36000000
> query_timeout_threshold=36000000
> 2. benchmark 配置见附件
> 约10.26小时
> 3. 写入完成执行flush



--
This message was sent by Atlassian Jira
(v8.20.10#820010)