You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "刘珍 (Jira)" <ji...@apache.org> on 2022/12/09 07:04:00 UTC
[jira] [Created] (IOTDB-5165) [ compaction ]
刘珍 created IOTDB-5165:
-------------------------
Summary: [ compaction ]
Key: IOTDB-5165
URL: https://issues.apache.org/jira/browse/IOTDB-5165
Project: Apache IoTDB
Issue Type: Bug
Components: Core/Compaction, mpp-cluster
Affects Versions: master branch, 1.0.0
Reporter: 刘珍
Assignee: 周沛辰
master 2022-12-09_a31441c
合并失败，报错
2022-12-09 14:21:46,728 [pool-43-IoTDB-Compaction-8] ERROR o.a.i.d.e.c.CompactionUtils:281 - root.test.g2_0 Device root.test.g2_0.d_82215 is overlapped between file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670420064671-7-0-0.tsfile, status: COMPACTING and file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670420083777-8-0-0.tsfile, status: COMPACTING, end time in file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670420064671-7-0-0.tsfile, status: COMPACTING is 1535568410000, start time in file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670420083777-8-0-0.tsfile, status: COMPACTING is 1535558420000
2022-12-09 14:21:46,729 [pool-43-IoTDB-Compaction-8] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:184 - Failed to pass compaction validation, source files is: [file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670444897033-4581-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670444947849-4590-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670444999954-4600-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445051784-4609-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445101595-4619-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445153290-4628-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445204996-4638-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445254210-4647-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445304094-4656-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445355765-4666-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445407476-4675-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445458633-4685-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445509050-4694-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445558911-4703-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445608483-4712-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445660518-4722-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445709301-4731-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445760226-4741-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445808744-4750-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445860031-4759-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445910381-4769-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445959576-4778-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446011018-4787-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446062483-4797-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446113893-4806-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446165533-4816-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446216536-4825-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446267529-4835-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446318504-4844-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446368398-4853-0-0.tsfile, status: COMPACTING], target files is [file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670444897033-4581-1-0.tsfile, status: CLOSED]
2022-12-09 14:21:46,732 [pool-43-IoTDB-Compaction-8] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:265 - root.test.g2_0-5 [Compaction] Meet errors in inner space compaction.
java.lang.RuntimeException: Failed to pass compaction validation
at org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask.doCompaction(InnerSpaceCompactionTask.java:188)
at org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask.start(AbstractCompactionTask.java:76)
at org.apache.iotdb.db.engine.compaction.CompactionWorker.run(CompactionWorker.java:66)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
测试流程
1. 私有云1期 172.16.2.3 8C32GB 1000GB HDD
数据备份在 m_1207_a0b2c8c_fast1下的data
如何生成这个数据备份：
IoTDB 配置文件
ConfigNode配置
MAX_HEAP_SIZE="2G"
DataNode配置
MAX_HEAP_SIZE="20G"
MAX_DIRECT_MEMORY_SIZE="6G"
Common配置（因为1207的版本的bug，合并全部失败，生成的tsfile全部是0层文件，所以可以关合并，生成数据）
time_partition_interval=60480000000
avg_series_point_number_threshold=10000
关合并
config_node_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
data_region_consensus_protocol_class=org.apache.iotdb.consensus.iot.IoTConsensus
启动集群start-standalone.sh
BM在172.16.2.27 4C16GB /data/iotdb/benchmark_v1
run.sh/run_conf.sh 放到/data/iotdb/benchmark_v1下
1.conf ~ 10.conf放到/data/iotdb/benchmark_v1/conf下，
启动BM 运行run.sh , 约运行10小时
2. 把data文件夹copy到master 1209 a31441c对应的路径
配置参数同步骤1的集群，区别在于需要打开合并，显式设置
cross_performer=fast
启动1副本1C1D集群
查看合并日志
日志备份在 ip3的/data1/iotdb/m_1209_a31441c_fast1/logs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)