You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "刘珍 (Jira)" <ji...@apache.org> on 2022/01/27 03:32:00 UTC
[jira] [Commented] (IOTDB-2471) [compaction] Pre-release 2 merges slower than pre-release 1
[ https://issues.apache.org/jira/browse/IOTDB-2471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482856#comment-17482856 ]
刘珍 commented on IOTDB-2471:
---------------------------
经过测试，打开乱序合并参数，可以提升合并速度（可能是适合我这个场景）：
enable_unseq_space_compaction=true
compaction_write_throughput_mb_per_sec=32
预发版2合并完成耗时：7小时20分钟
预发版1合并完成耗时：6小时
fix-not-release-write-lock-in-compaction分支：
enable_unseq_space_compaction=true
compaction_write_throughput_mb_per_sec=32
合并完成耗时：7小时30分钟
> [compaction] Pre-release 2 merges slower than pre-release 1
> -----------------------------------------------------------
>
> Key: IOTDB-2471
> URL: https://issues.apache.org/jira/browse/IOTDB-2471
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Server
> Affects Versions: 0.13.0-SNAPSHOT
> Reporter: 刘珍
> Assignee: Liuxuxin
> Priority: Major
> Attachments: pre1-iotdb-engine.properties, pre2-iotdb-engine.properties
>
>
> 问题描述:
> 0.13.0 预发版2 比预发版1 合并慢
> 1.数据备份相同，预发版1合并完成耗时6小时（合并完成时有290个tsfile）。
> 预发版2 已经合并了21个小时，还有17743个tsfile。
> 2.iotdb-env.sh相同
> 3.预发1的配置文件和预发版2有不同，配置文件见附件。
> 2个版本的配置参数均为：
> compaction_write_throughput_mb_per_sec=32
> 不同的配置参数有：
> 版本1 版本2
> enable_unseq_space_compaction=true # enable_unseq_space_compaction=false
> avg_series_point_number_threshold=100000 # avg_series_point_number_threshold=10000
> 4. 预发版2 刚启动起来之后，执行查询，有下面的错误信息：
> Msg: 500: [INTERNAL_SERVER_ERROR(500)] Exception occurred: "select count(*) from root.test.g_3.d_23". executeStatement failed. java.io.IOException: overlapped data should be consumed first
> 在2022-01-22 13:18:03,235 [pool-37-IoTDB-Query-3] INFO o.a.i.d.s.t.i.TSServiceImpl$QueryTask:233 - Cost: 327318 ms, sql is select count(*) from root.test.g_3.d_23
> 这个时间点再次执行同样的查询，有正确的结果。
> 2022-01-22 03:11:50,917 [pool-12-IoTDB-RPC-Client-41] WARN o.a.i.d.u.ErrorHandlingUtils:61 - Status code: INTERNAL_SERVER_ERROR(500), operation: "select count(*) from root.test.g_3.d_23". executeStatement failed
> java.util.concurrent.ExecutionException: java.io.IOException: overlapped data should be consumed first
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeStatement(TSServiceImpl.java:636)
> at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeStatement.getResult(TSIService.java:2793)
> at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeStatement.getResult(TSIService.java:2773)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: overlapped data should be consumed first
> at org.apache.iotdb.db.query.reader.series.SeriesReader.isPageOverlapped(SeriesReader.java:666)
> at org.apache.iotdb.db.query.reader.series.AlignedSeriesAggregateReader.canUseCurrentPageStatistics(AlignedSeriesAggregateReader.java:139)
> at org.apache.iotdb.db.query.executor.AggregationExecutor.aggregateAlignedPages(AggregationExecutor.java:547)
> at org.apache.iotdb.db.query.executor.AggregationExecutor.aggregateFromAlignedReader(AggregationExecutor.java:474)
> at org.apache.iotdb.db.query.executor.AggregationExecutor.aggregateOneAlignedSeries(AggregationExecutor.java:348)
> at org.apache.iotdb.db.query.executor.AggregationExecutor.aggregateOneAlignedSeries(AggregationExecutor.java:237)
> at org.apache.iotdb.db.query.executor.AggregationExecutor.executeWithoutValueFilter(AggregationExecutor.java:147)
> at org.apache.iotdb.db.query.executor.QueryRouter.aggregate(QueryRouter.java:158)
> at org.apache.iotdb.db.qp.executor.PlanExecutor.processDataQuery(PlanExecutor.java:622)
> at org.apache.iotdb.db.qp.executor.PlanExecutor.processQuery(PlanExecutor.java:250)
> at org.apache.iotdb.db.service.basic.ServiceProvider.createQueryDataSet(ServiceProvider.java:264)
> at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeQueryPlan(TSServiceImpl.java:829)
> at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.access$000(TSServiceImpl.java:162)
> at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl$QueryTask.call(TSServiceImpl.java:220)
> at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl$QueryTask.call(TSServiceImpl.java:164)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 common frames omitted
--
This message was sent by Atlassian Jira
(v8.20.1#820001)