You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "Jinrui Zhang (Jira)" <ji...@apache.org> on 2022/12/11 08:32:00 UTC
[jira] [Commented] (IOTDB-5165) [ compaction ]
[ https://issues.apache.org/jira/browse/IOTDB-5165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645744#comment-17645744 ]
Jinrui Zhang commented on IOTDB-5165:
-------------------------------------
Need to take a look in priority
> [ compaction ]
> --------------
>
> Key: IOTDB-5165
> URL: https://issues.apache.org/jira/browse/IOTDB-5165
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Compaction, mpp-cluster
> Affects Versions: master branch, 1.0.0
> Reporter: 刘珍
> Assignee: 周沛辰
> Priority: Major
> Attachments: 1.conf, 10.conf, 2.conf, 3.conf, 4.conf, 5.conf, 6.conf, 7.conf, 8.conf, 9.conf, run.sh, run_conf.sh
>
>
> master 2022-12-09_a31441c
> 合并失败，报错
> 2022-12-09 14:21:46,728 [pool-43-IoTDB-Compaction-8] ERROR o.a.i.d.e.c.CompactionUtils:281 - root.test.g2_0 Device root.test.g2_0.d_82215 {color:#DE350B}*is overlapped between file*{color} is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670420064671-7-0-0.tsfile, status: COMPACTING and file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670420083777-8-0-0.tsfile, status: COMPACTING, end time in file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670420064671-7-0-0.tsfile, status: COMPACTING is 1535568410000, start time in file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670420083777-8-0-0.tsfile, status: COMPACTING is 1535558420000
> 2022-12-09 14:21:46,729 [pool-43-IoTDB-Compaction-8] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:184 - {color:#DE350B}*Failed to pass compaction validation*{color}, source files is: [file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670444897033-4581-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670444947849-4590-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670444999954-4600-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445051784-4609-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445101595-4619-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445153290-4628-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445204996-4638-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445254210-4647-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445304094-4656-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445355765-4666-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445407476-4675-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445458633-4685-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445509050-4694-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445558911-4703-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445608483-4712-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445660518-4722-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445709301-4731-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445760226-4741-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445808744-4750-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445860031-4759-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445910381-4769-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445959576-4778-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446011018-4787-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446062483-4797-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446113893-4806-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446165533-4816-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446216536-4825-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446267529-4835-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446318504-4844-0-0.tsfile, status: COMPACTING, file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446368398-4853-0-0.tsfile, status: COMPACTING], target files is [file is /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670444897033-4581-1-0.tsfile, status: CLOSED]
> 2022-12-09 14:21:46,732 [pool-43-IoTDB-Compaction-8] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:265 - root.test.g2_0-5 [Compaction] Meet errors in inner space compaction.
> java.lang.RuntimeException: Failed to pass compaction validation
> at org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask.doCompaction(InnerSpaceCompactionTask.java:188)
> at org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask.start(AbstractCompactionTask.java:76)
> at org.apache.iotdb.db.engine.compaction.CompactionWorker.run(CompactionWorker.java:66)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 测试流程
> 1. 私有云1期 172.16.2.3 8C32GB 1000GB HDD
> 数据备份在 m_1207_a0b2c8c_fast1下的data
> 如何生成这个数据备份：
> IoTDB 配置文件
> ConfigNode配置
> MAX_HEAP_SIZE="2G"
> DataNode配置
> MAX_HEAP_SIZE="20G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> Common配置（因为1207的版本的bug，合并全部失败，生成的tsfile全部是0层文件，所以可以关合并，生成数据）
> time_partition_interval=60480000000
> avg_series_point_number_threshold=10000
> 关合并
> config_node_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.iot.IoTConsensus
> 启动集群start-standalone.sh
> BM在172.16.2.27 4C16GB /data/iotdb/benchmark_v1
> run.sh/run_conf.sh 放到/data/iotdb/benchmark_v1下
> 1.conf ~ 10.conf放到/data/iotdb/benchmark_v1/conf下，
> 启动BM 运行run.sh , 约运行10小时
> 2. 把data文件夹copy到master 1209 a31441c对应的路径
> 配置参数同步骤1的集群，区别在于需要打开合并，显式设置
> cross_performer=fast
> 启动1副本1C1D集群
> 查看合并日志
> 日志备份在 ip3的/data1/iotdb/m_1209_a31441c_fast1/logs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)