You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "周沛辰 (Jira)" <ji...@apache.org> on 2023/05/08 12:20:00 UTC
[jira] [Commented] (IOTDB-5836) [Compaction] Meet errors in cross space compaction.Caused by: java.nio.BufferUnderflowException: null
[ https://issues.apache.org/jira/browse/IOTDB-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720486#comment-17720486 ]
周沛辰 commented on IOTDB-5836:
----------------------------
获取到合并报错的文件，是顺序空间内合并了一层的顺序文件，使用TsFileSequenceRead工具进行排查，发现该文件里总共有200个设备，每个设备有12个序列，其中有两条序列存在解码时报BufferUnderFlow异常：
# 序列 root.test.g_0.d_404.s_4 在该文件里同一个chunkgroup里的第二个 chunk里的第一个page里第1024个点 报错：
!image-2023-05-08-20-17-31-168.png|width=487,height=333!
!image-2023-05-08-20-18-11-646.png|width=582,height=537!
2. 序列 root.test.g_0.d_599.s_4 在该文件里同一个chunkgroup里的第二个 chunk里的第21个page里第1024个点 报错：
!image-2023-05-08-20-19-19-673.png|width=497,height=289!
!image-2023-05-08-20-19-33-158.png|width=535,height=509!
> [Compaction] Meet errors in cross space compaction.Caused by: java.nio.BufferUnderflowException: null
> -----------------------------------------------------------------------------------------------------
>
> Key: IOTDB-5836
> URL: https://issues.apache.org/jira/browse/IOTDB-5836
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Compaction, mpp-cluster
> Affects Versions: 1.1.0
> Reporter: 刘珍
> Assignee: 周沛辰
> Priority: Major
> Attachments: 0427_aligned_compress.conf, 0427_aligned_compress_unseq.conf, image-2023-05-08-20-17-31-168.png, image-2023-05-08-20-18-11-646.png, image-2023-05-08-20-19-19-673.png, image-2023-05-08-20-19-33-158.png, insert.sh
>
>
> 测试版本： iotdb master 0427_ed8d539
> 日志路径：192.168.130.2 16C32G /data2/i_m_0427_ed8d539/logs
> 问题描述：
> 2023-04-27 19:51:27,391 [pool-16-IoTDB-Compaction-1] ERROR o.a.i.d.e.c.e.t.CrossSpaceCompactionTask:251 - root.test.g_0-2 [Compaction] Meet errors in cross space compaction.
> java.lang.RuntimeException: java.nio.BufferUnderflowException
> at org.apache.iotdb.db.engine.compaction.execute.performer.impl.FastCompactionPerformer.perform(FastCompactionPerformer.java:152)
> at org.apache.iotdb.db.engine.compaction.execute.task.CrossSpaceCompactionTask.doCompaction(CrossSpaceCompactionTask.java:160)
> at org.apache.iotdb.db.engine.compaction.execute.task.AbstractCompactionTask.start(AbstractCompactionTask.java:80)
> at org.apache.iotdb.db.engine.compaction.schedule.CompactionWorker.run(CompactionWorker.java:65)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.BufferUnderflowException: null
> at java.nio.Buffer.nextGetIndex(Buffer.java:524)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:138)
> at org.apache.iotdb.tsfile.utils.BitReader.next(BitReader.java:67)
> at org.apache.iotdb.tsfile.utils.BitReader.next(BitReader.java:36)
> at org.apache.iotdb.tsfile.encoding.decoder.FreqDecoder.decodeIndex(FreqDecoder.java:129)
> at org.apache.iotdb.tsfile.encoding.decoder.FreqDecoder.loadBlock(FreqDecoder.java:86)
> at org.apache.iotdb.tsfile.encoding.decoder.FreqDecoder.readDouble(FreqDecoder.java:45)
> at org.apache.iotdb.tsfile.read.reader.page.ValuePageReader.writeColumnBuilderWithNextBatch(ValuePageReader.java:306)
> at org.apache.iotdb.tsfile.read.reader.page.AlignedPageReader.getAllSatisfiedData(AlignedPageReader.java:226)
> at org.apache.iotdb.tsfile.read.reader.chunk.AlignedChunkReader.readPageData(AlignedChunkReader.java:320)
> at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.element.PageElement.deserializePage(PageElement.java:96)
> at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.element.PointElement.<init>(PointElement.java:35)
> at org.apache.iotdb.db.engine.compaction.execute.utils.reader.PointPriorityReader.addNewPage(PointPriorityReader.java:175)
> at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.checkAndCompactOverlapPage(SeriesCompactionExecutor.java:341)
> at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.compactWithOverlapPages(SeriesCompactionExecutor.java:307)
> at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.compactPages(SeriesCompactionExecutor.java:236)
> at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.compactWithOverlapChunks(SeriesCompactionExecutor.java:167)
> at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.compactChunks(SeriesCompactionExecutor.java:147)
> at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.AlignedSeriesCompactionExecutor.compactFiles(AlignedSeriesCompactionExecutor.java:97)
> at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.AlignedSeriesCompactionExecutor.execute(AlignedSeriesCompactionExecutor.java:84)
> at org.apache.iotdb.db.engine.compaction.execute.task.subtask.FastCompactionPerformerSubTask.call(FastCompactionPerformerSubTask.java:141)
> at org.apache.iotdb.db.engine.compaction.execute.performer.impl.FastCompactionPerformer.compactAlignedSeries(FastCompactionPerformer.java:199)
> at org.apache.iotdb.db.engine.compaction.execute.performer.impl.FastCompactionPerformer.perform(FastCompactionPerformer.java:133)
> ... 8 common frames omitted
> 测试流程
> 1. start-standalone.sh 启动1C1D
> confignode-env.sh
> MAX_HEAP_SIZE="2G"
> datanode-env.sh
> MAX_HEAP_SIZE="16G"
> MAX_DIRECT_MEMORY_SIZE="4G"
> iotdb-common.properties
> series_slot_num=1000
> default_storage_group_level=2
> 2.创建2个dev，12序列/dev，DOUBLE类型，指定不同的压缩方式
> 不同编码：
> dev定义如下：
> create database root.test.g_0;
> create ALIGNED timeseries root.test.g_0.d_0(s_0 DOUBLE compressor=SNAPPY,s_1 DOUBLE compressor=LZ4,s_2 DOUBLE compressor=GZIP,s_3 DOUBLE compressor=ZSTD,s_4 DOUBLE compressor=LZMA2,s_5 DOUBLE compressor=UNCOMPRESSED,s_6 DOUBLE compressor=LZ4,s_7 DOUBLE compressor=GZIP,s_8 DOUBLE compressor=ZSTD,s_9 DOUBLE compressor=LZMA2,s_10 DOUBLE compressor=SNAPPY,s_11 DOUBLE compressor=UNCOMPRESSED);
> create ALIGNED timeseries root.test.g_0.d_1(s_0 DOUBLE compressor=SNAPPY,s_1 DOUBLE ENCODING=RLE compressor=LZ4,s_2 DOUBLE ENCODING=TS_2DIFF compressor=GZIP,s_3 DOUBLE ENCODING=GORILLA compressor=ZSTD,s_4 DOUBLE ENCODING=FREQ compressor=LZMA2,s_5 DOUBLE ENCODING=CHIMP compressor=UNCOMPRESSED,s_6 DOUBLE ENCODING=SPRINTZ compressor=LZ4,s_7 DOUBLE ENCODING=RLBE compressor=GZIP,s_8 DOUBLE compressor=ZSTD,s_9 DOUBLE compressor=LZMA2,s_10 DOUBLE compressor=SNAPPY,s_11 DOUBLE compressor=UNCOMPRESSED)
> 每个序列写入960个点。见insert.sh 先写d_0，再写d_1,需要编辑一下脚本中的dev名字。
> 3. 再创建998个dev，用bm写入数据
> 运行如下脚本，创建dev
> #!/bin/bash
> for i in {2..999}
> do
> ./sbin/start-cli.sh -e "create ALIGNED timeseries root.test.g_0.d_$i(s_0 DOUBLE compressor=SNAPPY,\
> s_1 DOUBLE ENCODING=RLE compressor=LZ4,\
> s_2 DOUBLE ENCODING=TS_2DIFF compressor=GZIP,\
> s_3 DOUBLE ENCODING=GORILLA compressor=ZSTD,\
> s_4 DOUBLE ENCODING=FREQ compressor=LZMA2,\
> s_5 DOUBLE ENCODING=CHIMP compressor=UNCOMPRESSED,\
> s_6 DOUBLE ENCODING=SPRINTZ compressor=LZ4,\
> s_7 DOUBLE ENCODING=RLBE compressor=GZIP,\
> s_8 DOUBLE compressor=ZSTD,\
> s_9 DOUBLE compressor=LZMA2,\
> s_10 DOUBLE compressor=SNAPPY,\
> s_11 DOUBLE compressor=UNCOMPRESSED) "
> done
> 运行附件中的配置，写入数据：
> 先运行：0427_aligned_compress.conf
> 再运行：0427_aligned_compress_unseq.conf
> 大概2小时30分钟后，出现上述异常。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)