You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iotdb.apache.org by su...@apache.org on 2022/01/19 06:01:42 UTC
[iotdb] 01/01: Split large TsFile tool
This is an automated email from the ASF dual-hosted git repository.
sunzesong pushed a commit to branch jira-2078-rel/0.12
in repository https://gitbox.apache.org/repos/asf/iotdb.git
commit 1d39d1f130d7180aabbd5bbd57c610b62427ef75
Author: Zesong Sun <v-...@microsoft.com>
AuthorDate: Wed Jan 19 14:00:00 2022 +0800
Split large TsFile tool
---
docs/UserGuide/System-Tools/TsFile-Split-Tool.md | 51 ++++
.../zh/UserGuide/System-Tools/TsFile-Split-Tool.md | 50 ++++
.../tools/tsfileToolSet/split-tsfile-tool.bat | 62 +++++
.../tools/tsfileToolSet/split-tsfile-tool.sh | 48 ++++
.../org/apache/iotdb/db/tools/TsFileSplitTool.java | 293 +++++++++++++++++++++
.../iotdb/tsfile/read/TsFileSequenceReader.java | 88 +++++++
6 files changed, 592 insertions(+)
diff --git a/docs/UserGuide/System-Tools/TsFile-Split-Tool.md b/docs/UserGuide/System-Tools/TsFile-Split-Tool.md
new file mode 100644
index 0000000..1df4017
--- /dev/null
+++ b/docs/UserGuide/System-Tools/TsFile-Split-Tool.md
@@ -0,0 +1,51 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# TsFile Split Tool
+
+IoTDB version 0.12 could produce large files, which leads to difficulties in maintenance. Therefore, since version 0.12.5 and 0.13, TsFile split tool is provided. The split tool can split the large TsFile into several small TsFile according to the configuration.
+
+After building the server, the startup script of this tool will appear under the `server\target\iotdb-server-{version}\tools\tsfileToolSet` directory.
+
+Command:
+
+For Windows:
+
+```
+.\split-tsfile-tool.bat <path of your TsFile> (-level <LEVEL of the target files>) (-size <SIZE of the target files>)
+```
+
+For Linux or MacOs:
+
+```
+./split-tsfile-tool.sh <path of your TsFile> (-level <LEVEL of the target files>) (-size <SIZE of the target files>)
+```
+
+> Note that if `-level` is not set, the default level of target files is 10; if `-size` is not set, the default size of target files is about 1GB.
+
+Here are some configurations:
+
+1. The size of target files could be configured by the input param, which is 1GB by default. This configuration is also the target file size in compaction in 0.13. File size could determine whether the compaction is proceeded in 0.13, so this configuration could make sure there is no compaction after restarting.
+2. The level of target files is determined by the input param, which is 10 by default. File level could determine whether the compaction is proceeded in 0.12, so this configuration could make sure there is no compaction after restarting.
+
+Here are some more tips:
+1. TsFile split tool is offline maintenance tool. Before splitting a file, you should make sure the file to be split is closed (aka with `tsFile.resource`) and IoTDB is shut down. After splitting, restart IoTDB.
+2. TsFile split tool doesn't support splitting TsFile with deletion interval (aka with `.mods` file) and with aligned timeseries.
\ No newline at end of file
diff --git a/docs/zh/UserGuide/System-Tools/TsFile-Split-Tool.md b/docs/zh/UserGuide/System-Tools/TsFile-Split-Tool.md
new file mode 100644
index 0000000..93a9acb
--- /dev/null
+++ b/docs/zh/UserGuide/System-Tools/TsFile-Split-Tool.md
@@ -0,0 +1,50 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# TsFile 拆分工具
+
+0.12 版本的 IoTDB 会产生很大的文件，在运维过程中分析起来比较困难。因此，从 0.12.5 版本和 0.13 版本起，提供TsFile 分离工具，该工具可以将大的 TsFile 文件根据配置项拆分为数个小文件。该启动脚本会在编译 server 之后生成至 `server\target\iotdb-server-{version}\tools\tsfileToolSet` 目录中。
+
+使用方式：
+
+Windows:
+
+```
+.\split-tsfile-tool.bat <TsFile 文件路径> (-level <所拆分文件的层级>) (-size <所拆分文件的大小>)
+```
+
+
+Linux or MacOs:
+
+```
+./split-tsfile-tool.bat <TsFile 文件路径> (-level <所拆分文件的层级>) (-size <所拆分文件的大小>)
+```
+> 注意：如果不传入-level，所拆分文件的层级为 10；如果不传入-size，所拆分文件的大小约为 1GB；
+
+拆分中可以调节的配置项如下：
+
+1. 拆分所生成的文件大小通过命令传入参数确定的，默认为 1GB。这个配置项同样也是 0.13 版本中合并所能生成文件的目标大小。在 0.13 版本中，文件是否可以合并是通过文件大小确定的，可以通过此配置项控制重启后不继续合并。
+2. 文件所在层级是通过命令传入参数确定的，默认为 10。在 0.12 版本中，文件是否可以合并是通过文件所在层级确定的，可以通过此配置项控制重启后不继续合并。
+
+使用拆分工具需要注意如下事项：
+
+1. 拆分工具为离线运维工具，使用前需关闭 IoTDB，确保所拆分的文件已经完全落盘（即有`tsFile.resource`文件）。拆分后需移除原文件后重启。
+2. 拆分工具目前尚不支持拆分带有删除区间的 TsFile（即有`.mods`文件）和写有对齐时间序列的 TsFile。
\ No newline at end of file
diff --git a/server/src/assembly/resources/tools/tsfileToolSet/split-tsfile-tool.bat b/server/src/assembly/resources/tools/tsfileToolSet/split-tsfile-tool.bat
new file mode 100644
index 0000000..5e78cb3
--- /dev/null
+++ b/server/src/assembly/resources/tools/tsfileToolSet/split-tsfile-tool.bat
@@ -0,0 +1,62 @@
+@REM
+@REM Licensed to the Apache Software Foundation (ASF) under one
+@REM or more contributor license agreements. See the NOTICE file
+@REM distributed with this work for additional information
+@REM regarding copyright ownership. The ASF licenses this file
+@REM to you under the Apache License, Version 2.0 (the
+@REM "License"); you may not use this file except in compliance
+@REM with the License. You may obtain a copy of the License at
+@REM
+@REM http://www.apache.org/licenses/LICENSE-2.0
+@REM
+@REM Unless required by applicable law or agreed to in writing,
+@REM software distributed under the License is distributed on an
+@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+@REM KIND, either express or implied. See the License for the
+@REM specific language governing permissions and limitations
+@REM under the License.
+@REM
+
+
+@echo off
+echo ````````````````````````
+echo Start Splitting TsFile
+echo ````````````````````````
+
+if "%OS%" == "Windows_NT" setlocal
+
+pushd %~dp0..\..
+if NOT DEFINED IOTDB_HOME set IOTDB_HOME=%CD%
+popd
+
+if NOT DEFINED MAIN_CLASS set MAIN_CLASS=org.apache.iotdb.db.tools.TsFileSplitTool
+if NOT DEFINED JAVA_HOME goto :err
+
+@REM -----------------------------------------------------------------------------
+@REM ***** CLASSPATH library setting *****
+@REM Ensure that any user defined CLASSPATH variables are not used on startup
+set CLASSPATH="%IOTDB_HOME%\lib\*"
+
+goto okClasspath
+
+:append
+set CLASSPATH=%CLASSPATH%;%1
+goto :eof
+
+@REM -----------------------------------------------------------------------------
+:okClasspath
+
+"%JAVA_HOME%\bin\java" -cp "%CLASSPATH%" %MAIN_CLASS% %*
+
+goto finally
+
+
+:err
+echo JAVA_HOME environment variable must be set!
+pause
+
+
+@REM -----------------------------------------------------------------------------
+:finally
+
+ENDLOCAL
\ No newline at end of file
diff --git a/server/src/assembly/resources/tools/tsfileToolSet/split-tsfile-tool.sh b/server/src/assembly/resources/tools/tsfileToolSet/split-tsfile-tool.sh
new file mode 100644
index 0000000..24e4ead
--- /dev/null
+++ b/server/src/assembly/resources/tools/tsfileToolSet/split-tsfile-tool.sh
@@ -0,0 +1,48 @@
+#!/bin/sh
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+echo ---------------------
+echo Starting Splitting TsFile
+echo ---------------------
+
+if [ -z "${IOTDB_HOME}" ]; then
+ export IOTDB_HOME="$(cd "`dirname "$0"`"/../..; pwd)"
+fi
+
+if [ -n "$JAVA_HOME" ]; then
+ for java in "$JAVA_HOME"/bin/amd64/java "$JAVA_HOME"/bin/java; do
+ if [ -x "$java" ]; then
+ JAVA="$java"
+ break
+ fi
+ done
+else
+ JAVA=java
+fi
+
+CLASSPATH=""
+for f in ${IOTDB_HOME}/lib/*.jar; do
+ CLASSPATH=${CLASSPATH}":"$f
+done
+
+MAIN_CLASS=org.apache.iotdb.db.tools.TsFileSplitTool
+
+"$JAVA" -cp "$CLASSPATH" "$MAIN_CLASS" "$@"
+exit $?
\ No newline at end of file
diff --git a/server/src/main/java/org/apache/iotdb/db/tools/TsFileSplitTool.java b/server/src/main/java/org/apache/iotdb/db/tools/TsFileSplitTool.java
new file mode 100644
index 0000000..cae3aa3
--- /dev/null
+++ b/server/src/main/java/org/apache/iotdb/db/tools/TsFileSplitTool.java
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.tools;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+import org.apache.iotdb.tsfile.encoding.decoder.Decoder;
+import org.apache.iotdb.tsfile.exception.write.UnSupportedDataTypeException;
+import org.apache.iotdb.tsfile.file.MetaMarker;
+import org.apache.iotdb.tsfile.file.header.ChunkHeader;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetadata;
+import org.apache.iotdb.tsfile.file.metadata.TimeseriesMetadata;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.reader.page.PageReader;
+import org.apache.iotdb.tsfile.utils.Binary;
+import org.apache.iotdb.tsfile.write.chunk.ChunkWriterImpl;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+import org.apache.iotdb.tsfile.write.writer.TsFileIOWriter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class TsFileSplitTool {
+
+ private static final Logger logger = LoggerFactory.getLogger(TsFileSplitTool.class);
+
+ private final String filename;
+
+ private static final String SIZE_PARAM = "-size";
+ private static final String LEVEL_PARAM = "-level";
+
+ /**
+ * If the chunk point num is lower than this threshold, it will be deserialized into points,
+ * default is 100
+ */
+ private static final long chunkPointNumLowerBoundInCompaction = 100;
+
+ /** The target tsfile size, 1 GB by default */
+ private static long targetSplitFileSize = 1073741824L;
+
+ private static String levelNum = "10";
+
+ private static final FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ // TODO maxPlanIndex and minPlanIndex should be modified after cluster is refactored
+ // Maximum index of plans executed within this TsFile
+ protected long maxPlanIndex = Long.MIN_VALUE;
+
+ // Minimum index of plans executed within this TsFile.
+ protected long minPlanIndex = Long.MAX_VALUE;
+
+ public static void main(String[] args) throws IOException {
+ checkArgs(args);
+ String fileName = args[0];
+ logger.info("Splitting TsFile {} ...", fileName);
+ new TsFileSplitTool(fileName).run();
+ }
+
+ /* construct TsFileSketchTool */
+ public TsFileSplitTool(String filename) {
+ this.filename = filename;
+ }
+
+ /* entry of tool */
+ @SuppressWarnings("squid:S3776") // Suppress high Cognitive Complexity warning
+ public void run() throws IOException {
+ if (fsFactory.getFile(filename + ModificationFile.FILE_SUFFIX).exists()) {
+ throw new IOException("Unsupported to split TsFile with modification currently.");
+ }
+
+ TsFileIOWriter writer = null;
+
+ try (TsFileSequenceReader reader = new TsFileSequenceReader(filename)) {
+ Iterator<List<Path>> pathIterator = reader.getPathsIterator();
+ Set<String> devices = new HashSet<>();
+ String[] filePathSplit = filename.split(IoTDBConstant.FILE_NAME_SEPARATOR);
+ int originVersionIndex = Integer.parseInt(filePathSplit[filePathSplit.length - 3]);
+ int versionIndex = originVersionIndex + 1;
+ filePathSplit[filePathSplit.length - 2] = levelNum;
+
+ while (pathIterator.hasNext()) {
+ for (Path path : pathIterator.next()) {
+ String deviceId = path.getDevice();
+ if (devices.add(deviceId)) {
+ if (writer != null && writer.getPos() < targetSplitFileSize) {
+ writer.endChunkGroup();
+ writer.startChunkGroup(deviceId);
+ } else {
+ if (writer != null) {
+ // seal last TsFile
+ TsFileResource resource = endFileAndGenerateResource(writer);
+ resource.close();
+ }
+
+ filePathSplit[filePathSplit.length - 3] = String.valueOf(versionIndex);
+ StringBuilder sb = new StringBuilder();
+ for (int i = 0; i < filePathSplit.length; i++) {
+ sb.append(filePathSplit[i]);
+ if (i != filePathSplit.length - 1) {
+ sb.append(IoTDBConstant.FILE_NAME_SEPARATOR);
+ }
+ }
+ // open a new TsFile
+ writer = new TsFileIOWriter(fsFactory.getFile(sb.toString()));
+ versionIndex++;
+ writer.startChunkGroup(deviceId);
+ }
+ }
+
+ List<ChunkMetadata> chunkMetadataList = reader.getChunkMetadataList(path);
+ assert writer != null;
+
+ ChunkMetadata firstChunkMetadata = chunkMetadataList.get(0);
+ reader.position(firstChunkMetadata.getOffsetOfChunkHeader());
+ ChunkHeader chunkHeader = reader.readChunkHeader(reader.readMarker());
+ MeasurementSchema measurementSchema =
+ new MeasurementSchema(
+ chunkHeader.getMeasurementID(), chunkHeader.getDataType(),
+ chunkHeader.getEncodingType(), chunkHeader.getCompressionType());
+
+ int numInChunk = 0;
+ ChunkWriterImpl chunkWriter = new ChunkWriterImpl(measurementSchema);
+
+ for (int i = 0; i < chunkMetadataList.size(); i++) {
+ if (i != 0) {
+ reader.position(chunkMetadataList.get(i).getOffsetOfChunkHeader());
+ chunkHeader = reader.readChunkHeader(reader.readMarker());
+ }
+
+ TSDataType dataType = chunkHeader.getDataType();
+ int dataSize = chunkHeader.getDataSize();
+ Decoder valueDecoder =
+ Decoder.getDecoderByType(chunkHeader.getEncodingType(), dataType);
+ Decoder defaultTimeDecoder =
+ Decoder.getDecoderByType(
+ TSEncoding.valueOf(TSFileDescriptor.getInstance().getConfig().getTimeEncoder()),
+ TSDataType.INT64);
+
+ while (dataSize > 0) {
+ valueDecoder.reset();
+
+ // a new Page
+ PageHeader pageHeader =
+ reader.readPageHeader(
+ dataType,
+ ((byte) (chunkHeader.getChunkType() & 0x3F)) == MetaMarker.CHUNK_HEADER);
+ ByteBuffer pageData = reader.readPage(pageHeader, chunkHeader.getCompressionType());
+ PageReader pageReader =
+ new PageReader(pageData, dataType, valueDecoder, defaultTimeDecoder, null);
+ BatchData batchData = pageReader.getAllSatisfiedPageData();
+
+ while (batchData.hasCurrent()) {
+ writeToChunkWriter(
+ chunkWriter,
+ batchData.currentTime(),
+ batchData.currentValue(),
+ chunkHeader.getDataType());
+ numInChunk++;
+ if (numInChunk == chunkPointNumLowerBoundInCompaction) {
+ chunkWriter.writeToFileWriter(writer);
+ numInChunk = 0;
+ }
+ batchData.next();
+ }
+
+ dataSize -= pageHeader.getSerializedPageSize();
+ }
+ }
+ if (numInChunk != 0) {
+ chunkWriter.writeToFileWriter(writer);
+ }
+ }
+ }
+
+ if (writer != null) {
+ // seal last TsFile
+ TsFileResource resource = endFileAndGenerateResource(writer);
+ resource.close();
+ }
+ } finally {
+ if (writer != null) {
+ writer.close();
+ }
+ }
+ }
+
+ private void writeToChunkWriter(
+ ChunkWriterImpl chunkWriter, long time, Object value, TSDataType dataType) {
+ switch (dataType) {
+ case INT32:
+ chunkWriter.write(time, (int) value);
+ break;
+ case INT64:
+ chunkWriter.write(time, (long) value);
+ break;
+ case FLOAT:
+ chunkWriter.write(time, (float) value);
+ break;
+ case DOUBLE:
+ chunkWriter.write(time, (double) value);
+ break;
+ case BOOLEAN:
+ chunkWriter.write(time, (boolean) value);
+ break;
+ case TEXT:
+ chunkWriter.write(time, (Binary) value);
+ break;
+ default:
+ throw new UnSupportedDataTypeException(
+ String.format("Data type %s is not supported.", dataType));
+ }
+ }
+
+ private TsFileResource endFileAndGenerateResource(TsFileIOWriter writer) throws IOException {
+ writer.endChunkGroup();
+ writer.endFile();
+
+ TsFileResource tsFileResource = new TsFileResource(writer.getFile());
+ Map<String, List<TimeseriesMetadata>> deviceTimeseriesMetadataMap =
+ writer.getDeviceTimeseriesMetadataMap();
+ for (Map.Entry<String, List<TimeseriesMetadata>> entry :
+ deviceTimeseriesMetadataMap.entrySet()) {
+ String device = entry.getKey();
+ for (TimeseriesMetadata timeseriesMetaData : entry.getValue()) {
+ tsFileResource.updateStartTime(device, timeseriesMetaData.getStatistics().getStartTime());
+ tsFileResource.updateEndTime(device, timeseriesMetaData.getStatistics().getEndTime());
+ }
+ }
+ tsFileResource.setMinPlanIndex(minPlanIndex);
+ tsFileResource.setMaxPlanIndex(maxPlanIndex);
+ tsFileResource.setClosed(true);
+ tsFileResource.serialize();
+
+ return tsFileResource;
+ }
+
+ private static void checkArgs(String[] args) {
+ if (args.length == 3) {
+ if (args[1].equals(SIZE_PARAM)) {
+ targetSplitFileSize = Long.parseLong(args[2]);
+ return;
+ } else if (args[1].equals(LEVEL_PARAM)) {
+ levelNum = args[2];
+ return;
+ }
+ } else if (args.length == 5) {
+ if (args[1].equals(SIZE_PARAM) && args[3].equals(LEVEL_PARAM)) {
+ targetSplitFileSize = Long.parseLong(args[2]);
+ levelNum = args[4];
+ return;
+ } else if (args[1].equals(LEVEL_PARAM) && args[3].equals(SIZE_PARAM)) {
+ levelNum = args[2];
+ targetSplitFileSize = Long.parseLong(args[4]);
+ return;
+ }
+ }
+ throw new UnsupportedOperationException("Invalid param");
+ }
+}
diff --git a/tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java b/tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java
index 3fb94f1..7759d4e 100644
--- a/tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java
+++ b/tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java
@@ -578,6 +578,94 @@ public class TsFileSequenceReader implements AutoCloseable {
}
/**
+ * @return an iterator of timeseries list, in which names of timeseries are ordered in dictionary
+ * order
+ * @throws IOException io error
+ */
+ public Iterator<List<Path>> getPathsIterator() throws IOException {
+ readFileMetadata();
+
+ MetadataIndexNode metadataIndexNode = tsFileMetaData.getMetadataIndex();
+ List<MetadataIndexEntry> metadataIndexEntryList = metadataIndexNode.getChildren();
+ Queue<Pair<String, Pair<Long, Long>>> queue = new LinkedList<>();
+ for (int i = 0; i < metadataIndexEntryList.size(); i++) {
+ MetadataIndexEntry metadataIndexEntry = metadataIndexEntryList.get(i);
+ long endOffset = metadataIndexNode.getEndOffset();
+ if (i != metadataIndexEntryList.size() - 1) {
+ endOffset = metadataIndexEntryList.get(i + 1).getOffset();
+ }
+ ByteBuffer buffer = readData(metadataIndexEntry.getOffset(), endOffset);
+ getAllPaths(metadataIndexEntry, buffer, null, metadataIndexNode.getNodeType(), queue);
+ }
+ return new Iterator<List<Path>>() {
+ @Override
+ public boolean hasNext() {
+ return !queue.isEmpty();
+ }
+
+ @Override
+ public List<Path> next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ Pair<String, Pair<Long, Long>> startEndPair = queue.remove();
+ List<Path> paths = new ArrayList<>();
+ try {
+ ByteBuffer nextBuffer = readData(startEndPair.right.left, startEndPair.right.right);
+ while (nextBuffer.hasRemaining()) {
+ paths.add(
+ new Path(
+ startEndPair.left,
+ TimeseriesMetadata.deserializeFrom(nextBuffer, false).getMeasurementId()));
+ }
+ return paths;
+ } catch (IOException e) {
+ throw new TsFileRuntimeException(
+ "Error occurred while reading a time series metadata block.");
+ }
+ }
+ };
+ }
+
+ private void getAllPaths(
+ MetadataIndexEntry metadataIndex,
+ ByteBuffer buffer,
+ String deviceId,
+ MetadataIndexNodeType type,
+ Queue<Pair<String, Pair<Long, Long>>> queue)
+ throws IOException {
+ try {
+ if (type.equals(MetadataIndexNodeType.LEAF_DEVICE)) {
+ deviceId = metadataIndex.getName();
+ }
+ MetadataIndexNode metadataIndexNode = MetadataIndexNode.deserializeFrom(buffer);
+ int metadataIndexListSize = metadataIndexNode.getChildren().size();
+ for (int i = 0; i < metadataIndexListSize; i++) {
+ long startOffset = metadataIndexNode.getChildren().get(i).getOffset();
+ long endOffset = metadataIndexNode.getEndOffset();
+ if (i != metadataIndexListSize - 1) {
+ endOffset = metadataIndexNode.getChildren().get(i + 1).getOffset();
+ }
+ if (metadataIndexNode.getNodeType().equals(MetadataIndexNodeType.LEAF_MEASUREMENT)) {
+ queue.add(new Pair<>(deviceId, new Pair<>(startOffset, endOffset)));
+ continue;
+ }
+ ByteBuffer nextBuffer =
+ readData(metadataIndexNode.getChildren().get(i).getOffset(), endOffset);
+ getAllPaths(
+ metadataIndexNode.getChildren().get(i),
+ nextBuffer,
+ deviceId,
+ metadataIndexNode.getNodeType(),
+ queue);
+ }
+ } catch (BufferOverflowException e) {
+ logger.error("Something error happened while getting all paths of file {}", file);
+ throw e;
+ }
+ }
+
+ /**
* Traverse the metadata index from MetadataIndexEntry to get TimeseriesMetadatas
*
* @param metadataIndex MetadataIndexEntry