You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2021/07/26 09:27:34 UTC
[GitHub] [iotdb] Cpaulyz opened a new pull request #3627: [IOTDB-1517] Refactor TsFile Index for Vector(multi-variable timeseries)
Cpaulyz opened a new pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627
### TsFileSketchTool
Add new features.
* for*MetadataIndex*, show index tree to facilitate TsFile debug.
e.g.
```
| [MetadataIndex:LEAF_DEVICE]
| └───────[root.sg_1.d1,2485]
| [MetadataIndex:LEAF_MEASUREMENT]
| └───────[s2,1742]
| └───────[s6,1955]
| └───────[vector.s2,2139]
| └───────[root.sg_1.d2,2535]
| [MetadataIndex:LEAF_MEASUREMENT]
| └───────[s2,2201]
| └───────[s6,2414]
```
* for *ChunkMatadataList*, show device info.
* old version: `vector.s1`
* new version: ` (root.sg_1.d1.)`
* for *Chunk*, show more message such as chunk header, page header.
### MetadataIndexConstructor
* **Construct**
check according to [design document in confluence](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=184617773#TsFile%E6%96%87%E4%BB%B6%E6%A0%BC%E5%BC%8F-1.2.3.3IndexOfTimeseriesIndex%E6%97%B6%E9%97%B4%E5%BA%8F%E5%88%97%E7%B4%A2%E5%BC%95%E7%9A%84%E7%B4%A2%E5%BC%95%EF%BC%88%E4%BA%8C%E7%BA%A7%E7%B4%A2%E5%BC%95%EF%BC%89) example 5-7
* **Test**
* add test about metadata index tree in `MetadataIndexConstructorTest.java`
* to add more test case, you only need to create params `devices, vectorMeasurement, measurements` and call funtion `private void test(String[] devices, int[][] vectorMeasurement, String[][] singleMeasurement)`
* `test` funtion will automatically test by the following steps. Note that if the metadata index tree is re-designed, you may need to modify "generate correct result" in `generateCorrectResult`.
* ```
// 1. generate file
// 2. read metadata from file
// 3. generate correct result
// 4. compare correct result with TsFile's metadata
// 4.1 make sure device in order
// 4.2 make sure timeseries in order
// 4.3 make sure split leaf correctly
```
### Read
* **motivation:** New design of vector structure may split its value entry into different leaf node.
* **Problem:** However, query process reads subSensors accoring to the vector. As a result, in `TimeseriesMetadataCache.java`, funtion
```
public List<TimeseriesMetadata> get(
TimeSeriesMetadataCacheKey key,
List<String> subSensorList,
Set<String> allSensors,
boolean debug)
```
read TimeseriesMetadataList by vector.
For example, there is a vector `v1.(s1,s2)`, one leaf node stores [..., v1, v1.s1] and next node stores [v1.s2, ...]. Old implemtation call `reader.readTimeseriesMetadata(path, allSensors)`, which only read the leaf where time column stored. It may ignore the entry stored in other leaves such as v1.s2's entry.
* **Solution:** Extraction a common function `readTimeseriesMetadataForVector`, which reads the all timeseriesMetadata cover query key and subSensorList.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #3627: [IOTDB-1517] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
HTHou commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r676453435
##########
File path: session/src/test/java/org/apache/iotdb/session/IoTDBSessionVectorIT.java
##########
@@ -0,0 +1,201 @@
+package org.apache.iotdb.session;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.utils.EnvironmentUtils;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.schema.IMeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.VectorMeasurementSchema;
+
Review comment:
Apache header....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r678830547
##########
File path: tsfile/src/test/java/org/apache/iotdb/tsfile/write/MetadataIndexConstructorTest.java
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.tsfile.write;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.constant.TestConstant;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexEntry;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexNode;
+import org.apache.iotdb.tsfile.file.metadata.TimeseriesMetadata;
+import org.apache.iotdb.tsfile.file.metadata.TsFileMetadata;
+import org.apache.iotdb.tsfile.file.metadata.enums.MetadataIndexNodeType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.record.TSRecord;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.record.datapoint.DataPoint;
+import org.apache.iotdb.tsfile.write.record.datapoint.LongDataPoint;
+import org.apache.iotdb.tsfile.write.schema.IMeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.Schema;
+import org.apache.iotdb.tsfile.write.schema.VectorMeasurementSchema;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.shaded.org.apache.commons.lang.text.StrBuilder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.*;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** test for MetadataIndexConstructor */
+public class MetadataIndexConstructorTest {
+ private static final Logger logger = LoggerFactory.getLogger(MetadataIndexConstructorTest.class);
+ private final TSFileConfig conf = TSFileDescriptor.getInstance().getConfig();
+ private static final String FILE_PATH =
+ TestConstant.BASE_OUTPUT_PATH.concat("MetadataIndexConstructorTest.tsfile");
+
+ private static final String measurementPrefix = "sensor_";
+ private static final String vectorPrefix = "vector_";
+ private int maxDegreeOfIndexNode;
+
+ @Before
+ public void before() {
+ maxDegreeOfIndexNode = conf.getMaxDegreeOfIndexNode();
+ conf.setMaxDegreeOfIndexNode(10);
+ }
+
+ @After
+ public void after() {
+ conf.setMaxDegreeOfIndexNode(maxDegreeOfIndexNode);
+ }
+
+ /** Example 1: 5 entities with 5 measurements each */
+ @Test
+ public void singleIndexTest1() {
+ int deviceNum = 5;
+ int measurementNum = 5;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 2: 1 entity with 150 measurements */
+ @Test
+ public void singleIndexTest2() {
+ int deviceNum = 1;
+ int measurementNum = 150;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 3: 150 entities with 1 measurement each */
+ @Test
+ public void singleIndexTest3() {
+ int deviceNum = 150;
+ int measurementNum = 1;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + generateIndexString(i, deviceNum);
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 4: 150 entities with 150 measurements each */
+ @Test
+ public void singleIndexTest4() {
+ int deviceNum = 150;
+ int measurementNum = 1;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 5: 1 entities with 2 vectors, 9 measurements for each vector */
+ @Test
+ public void vectorIndexTest1() {
+ String[] devices = {"d0"};
+ int[][] vectorMeasurement = {{9, 9}};
+ test(devices, vectorMeasurement, null);
+ }
+
+ /** Example 6: 1 entities with 2 vectors, 15 measurements for each vector */
+ @Test
+ public void vectorIndexTest2() {
+ String[] devices = {"d0"};
+ int[][] vectorMeasurement = {{15, 15}};
+ test(devices, vectorMeasurement, null);
+ }
+
+ /**
+ * Example 7: 2 entities, measurements of entities are shown in the following table
+ *
+ * <p>d0.s0~s4 | d0.v0.(s0~s8) | d0.z0~z3 d1.s0~s14 | d1.v0.(s0~s3)
+ */
+ @Test
+ public void compositeIndexTest() {
+ String[] devices = {"d0", "d1"};
+ int[][] vectorMeasurement = {{9}, {4}};
+ String[][] singleMeasurement = {
+ {"s0", "s1", "s2", "s3", "s4", "z0", "z1", "z2", "z3"},
+ {
+ "s00", "s01", "s02", "s03", "s04", "s05", "s06", "s07", "s08", "s09", "s10", "s11", "s12",
+ "s13", "s14"
+ }
+ };
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /**
+ * start test
+ *
+ * @param devices name and number of device
+ * @param vectorMeasurement the number of device and the number of values to include in the tablet
+ * @param singleMeasurement non-vector measurement name, set null if no need
+ */
+ private void test(String[] devices, int[][] vectorMeasurement, String[][] singleMeasurement) {
+ // 1. generate file
+ generateFile(devices, vectorMeasurement, singleMeasurement);
+ // 2. read metadata from file
+ List<String> actualDevices = new ArrayList<>(); // contains all device by sequence
+ List<List<String>> actualMeasurements =
+ new ArrayList<>(); // contains all measurements group by device
+ readMetaDataDFS(actualDevices, actualMeasurements);
+ // 3. generate correct result
+ List<String> correctDevices = new ArrayList<>(); // contains all device by sequence
+ List<List<String>> correctFirstMeasurements =
+ new ArrayList<>(); // contains first measurements of every leaf, group by device
+ generateCorrectResult(
+ correctDevices, correctFirstMeasurements, devices, vectorMeasurement, singleMeasurement);
+ // 4. compare correct result with TsFile's metadata
+ Arrays.sort(devices);
+ // 4.1 make sure device in order
+ assertEquals(correctDevices.size(), devices.length);
+ assertEquals(actualDevices.size(), correctDevices.size());
+ for (int i = 0; i < actualDevices.size(); i++) {
+ assertEquals(actualDevices.get(i), correctDevices.get(i));
+ }
+ // 4.2 make sure timeseries in order
+ try (TsFileSequenceReader reader = new TsFileSequenceReader(FILE_PATH)) {
+ Map<String, List<TimeseriesMetadata>> allTimeseriesMetadata =
+ reader.getAllTimeseriesMetadata();
+ for (int j = 0; j < actualDevices.size(); j++) {
+ for (int i = 0; i < actualMeasurements.get(j).size(); i++) {
+ assertEquals(
+ allTimeseriesMetadata.get(actualDevices.get(j)).get(i).getMeasurementId(),
+ correctFirstMeasurements.get(j).get(i));
+ }
+ }
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ // 4.3 make sure split leaf correctly
+ for (int j = 0; j < actualDevices.size(); j++) {
+ for (int i = 0; i < actualMeasurements.get(j).size(); i++) {
+ assertEquals(
+ actualMeasurements.get(j).get(i),
+ correctFirstMeasurements.get(j).get(i * conf.getMaxDegreeOfIndexNode()));
+ }
+ }
+ }
+
+ /**
+ * read TsFile metadata, load actual message in devices and measurements
+ *
+ * @param devices load actual devices
+ * @param measurements load actual measurement(first of every leaf)
+ */
+ private void readMetaDataDFS(List<String> devices, List<List<String>> measurements) {
+ try (TsFileSequenceReader reader = new TsFileSequenceReader(FILE_PATH)) {
+ TsFileMetadata tsFileMetaData = reader.readFileMetadata();
+ MetadataIndexNode metadataIndexNode = tsFileMetaData.getMetadataIndex();
+ deviceDFS(devices, measurements, reader, metadataIndexNode);
+ } catch (IOException e) {
+ e.printStackTrace();
Review comment:
```suggestion
e.printStackTrace();
fail(e.getMessage());
```
##########
File path: tsfile/src/test/java/org/apache/iotdb/tsfile/write/MetadataIndexConstructorTest.java
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.tsfile.write;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.constant.TestConstant;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexEntry;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexNode;
+import org.apache.iotdb.tsfile.file.metadata.TimeseriesMetadata;
+import org.apache.iotdb.tsfile.file.metadata.TsFileMetadata;
+import org.apache.iotdb.tsfile.file.metadata.enums.MetadataIndexNodeType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.record.TSRecord;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.record.datapoint.DataPoint;
+import org.apache.iotdb.tsfile.write.record.datapoint.LongDataPoint;
+import org.apache.iotdb.tsfile.write.schema.IMeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.Schema;
+import org.apache.iotdb.tsfile.write.schema.VectorMeasurementSchema;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.shaded.org.apache.commons.lang.text.StrBuilder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.*;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** test for MetadataIndexConstructor */
+public class MetadataIndexConstructorTest {
+ private static final Logger logger = LoggerFactory.getLogger(MetadataIndexConstructorTest.class);
+ private final TSFileConfig conf = TSFileDescriptor.getInstance().getConfig();
+ private static final String FILE_PATH =
+ TestConstant.BASE_OUTPUT_PATH.concat("MetadataIndexConstructorTest.tsfile");
+
+ private static final String measurementPrefix = "sensor_";
+ private static final String vectorPrefix = "vector_";
+ private int maxDegreeOfIndexNode;
+
+ @Before
+ public void before() {
+ maxDegreeOfIndexNode = conf.getMaxDegreeOfIndexNode();
+ conf.setMaxDegreeOfIndexNode(10);
+ }
+
+ @After
+ public void after() {
+ conf.setMaxDegreeOfIndexNode(maxDegreeOfIndexNode);
+ }
+
+ /** Example 1: 5 entities with 5 measurements each */
+ @Test
+ public void singleIndexTest1() {
+ int deviceNum = 5;
+ int measurementNum = 5;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 2: 1 entity with 150 measurements */
+ @Test
+ public void singleIndexTest2() {
+ int deviceNum = 1;
+ int measurementNum = 150;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 3: 150 entities with 1 measurement each */
+ @Test
+ public void singleIndexTest3() {
+ int deviceNum = 150;
+ int measurementNum = 1;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + generateIndexString(i, deviceNum);
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 4: 150 entities with 150 measurements each */
+ @Test
+ public void singleIndexTest4() {
+ int deviceNum = 150;
+ int measurementNum = 1;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 5: 1 entities with 2 vectors, 9 measurements for each vector */
+ @Test
+ public void vectorIndexTest1() {
+ String[] devices = {"d0"};
+ int[][] vectorMeasurement = {{9, 9}};
+ test(devices, vectorMeasurement, null);
+ }
+
+ /** Example 6: 1 entities with 2 vectors, 15 measurements for each vector */
+ @Test
+ public void vectorIndexTest2() {
+ String[] devices = {"d0"};
+ int[][] vectorMeasurement = {{15, 15}};
+ test(devices, vectorMeasurement, null);
+ }
+
+ /**
+ * Example 7: 2 entities, measurements of entities are shown in the following table
+ *
+ * <p>d0.s0~s4 | d0.v0.(s0~s8) | d0.z0~z3 d1.s0~s14 | d1.v0.(s0~s3)
+ */
+ @Test
+ public void compositeIndexTest() {
+ String[] devices = {"d0", "d1"};
+ int[][] vectorMeasurement = {{9}, {4}};
+ String[][] singleMeasurement = {
+ {"s0", "s1", "s2", "s3", "s4", "z0", "z1", "z2", "z3"},
+ {
+ "s00", "s01", "s02", "s03", "s04", "s05", "s06", "s07", "s08", "s09", "s10", "s11", "s12",
+ "s13", "s14"
+ }
+ };
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /**
+ * start test
+ *
+ * @param devices name and number of device
+ * @param vectorMeasurement the number of device and the number of values to include in the tablet
+ * @param singleMeasurement non-vector measurement name, set null if no need
+ */
+ private void test(String[] devices, int[][] vectorMeasurement, String[][] singleMeasurement) {
+ // 1. generate file
+ generateFile(devices, vectorMeasurement, singleMeasurement);
+ // 2. read metadata from file
+ List<String> actualDevices = new ArrayList<>(); // contains all device by sequence
+ List<List<String>> actualMeasurements =
+ new ArrayList<>(); // contains all measurements group by device
+ readMetaDataDFS(actualDevices, actualMeasurements);
+ // 3. generate correct result
+ List<String> correctDevices = new ArrayList<>(); // contains all device by sequence
+ List<List<String>> correctFirstMeasurements =
+ new ArrayList<>(); // contains first measurements of every leaf, group by device
+ generateCorrectResult(
+ correctDevices, correctFirstMeasurements, devices, vectorMeasurement, singleMeasurement);
+ // 4. compare correct result with TsFile's metadata
+ Arrays.sort(devices);
+ // 4.1 make sure device in order
+ assertEquals(correctDevices.size(), devices.length);
+ assertEquals(actualDevices.size(), correctDevices.size());
+ for (int i = 0; i < actualDevices.size(); i++) {
+ assertEquals(actualDevices.get(i), correctDevices.get(i));
+ }
+ // 4.2 make sure timeseries in order
+ try (TsFileSequenceReader reader = new TsFileSequenceReader(FILE_PATH)) {
+ Map<String, List<TimeseriesMetadata>> allTimeseriesMetadata =
+ reader.getAllTimeseriesMetadata();
+ for (int j = 0; j < actualDevices.size(); j++) {
+ for (int i = 0; i < actualMeasurements.get(j).size(); i++) {
+ assertEquals(
+ allTimeseriesMetadata.get(actualDevices.get(j)).get(i).getMeasurementId(),
+ correctFirstMeasurements.get(j).get(i));
+ }
+ }
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ // 4.3 make sure split leaf correctly
+ for (int j = 0; j < actualDevices.size(); j++) {
+ for (int i = 0; i < actualMeasurements.get(j).size(); i++) {
+ assertEquals(
+ actualMeasurements.get(j).get(i),
+ correctFirstMeasurements.get(j).get(i * conf.getMaxDegreeOfIndexNode()));
+ }
+ }
+ }
+
+ /**
+ * read TsFile metadata, load actual message in devices and measurements
+ *
+ * @param devices load actual devices
+ * @param measurements load actual measurement(first of every leaf)
+ */
+ private void readMetaDataDFS(List<String> devices, List<List<String>> measurements) {
+ try (TsFileSequenceReader reader = new TsFileSequenceReader(FILE_PATH)) {
+ TsFileMetadata tsFileMetaData = reader.readFileMetadata();
+ MetadataIndexNode metadataIndexNode = tsFileMetaData.getMetadataIndex();
+ deviceDFS(devices, measurements, reader, metadataIndexNode);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ /** DFS in device level load actual devices */
+ private void deviceDFS(
+ List<String> devices,
+ List<List<String>> measurements,
+ TsFileSequenceReader reader,
+ MetadataIndexNode node) {
+ assertTrue(
+ node.getNodeType().equals(MetadataIndexNodeType.LEAF_DEVICE)
+ || node.getNodeType().equals(MetadataIndexNodeType.INTERNAL_DEVICE));
+ for (int i = 0; i < node.getChildren().size(); i++) {
+ MetadataIndexEntry metadataIndexEntry = node.getChildren().get(i);
+ long endOffset = node.getEndOffset();
+ if (i != node.getChildren().size() - 1) {
+ endOffset = node.getChildren().get(i + 1).getOffset();
+ }
+ MetadataIndexNode subNode =
+ reader.getMetadataIndexNode(metadataIndexEntry.getOffset(), endOffset);
+ if (node.getNodeType().equals(MetadataIndexNodeType.LEAF_DEVICE)) {
+ devices.add(metadataIndexEntry.getName());
+ measurements.add(new ArrayList<>());
+ measurementDFS(devices.size() - 1, measurements, reader, subNode);
+ } else if (node.getNodeType().equals(MetadataIndexNodeType.INTERNAL_DEVICE)) {
+ deviceDFS(devices, measurements, reader, subNode);
+ }
+ }
+ }
+ /** DFS in measurement level load actual measurements */
+ private void measurementDFS(
+ int deviceIndex,
+ List<List<String>> measurements,
+ TsFileSequenceReader reader,
+ MetadataIndexNode node) {
+ assertTrue(
+ node.getNodeType().equals(MetadataIndexNodeType.LEAF_MEASUREMENT)
+ || node.getNodeType().equals(MetadataIndexNodeType.INTERNAL_MEASUREMENT));
+ for (int i = 0; i < node.getChildren().size(); i++) {
+ MetadataIndexEntry metadataIndexEntry = node.getChildren().get(i);
+ long endOffset = node.getEndOffset();
+ if (i != node.getChildren().size() - 1) {
+ endOffset = node.getChildren().get(i + 1).getOffset();
+ }
+ if (node.getNodeType().equals(MetadataIndexNodeType.LEAF_MEASUREMENT)) {
+ // 把每个叶子节点的第一个加进来
+ measurements.get(deviceIndex).add(metadataIndexEntry.getName());
+ } else if (node.getNodeType().equals(MetadataIndexNodeType.INTERNAL_MEASUREMENT)) {
+ MetadataIndexNode subNode =
+ reader.getMetadataIndexNode(metadataIndexEntry.getOffset(), endOffset);
+ measurementDFS(deviceIndex, measurements, reader, subNode);
+ }
+ }
+ }
+
+ /**
+ * generate correct devices and measurements for test Note that if the metadata index tree is
+ * re-designed, you may need to modify this function as well.
+ *
+ * @param correctDevices output
+ * @param correctMeasurements output
+ * @param devices input
+ * @param vectorMeasurement input
+ * @param singleMeasurement input
+ */
+ private void generateCorrectResult(
+ List<String> correctDevices,
+ List<List<String>> correctMeasurements,
+ String[] devices,
+ int[][] vectorMeasurement,
+ String[][] singleMeasurement) {
+ for (int i = 0; i < devices.length; i++) {
+ String device = devices[i];
+ correctDevices.add(device);
+ // generate measurement and sort
+ List<String> measurements = new ArrayList<>();
+ // single-variable measurement
+ if (singleMeasurement != null) {
+ for (int j = 0; j < singleMeasurement[i].length; j++) {
+ measurements.add(singleMeasurement[i][j]);
+ }
+ }
+ // multi-variable measurement
+ for (int vectorIndex = 0; vectorIndex < vectorMeasurement[i].length; vectorIndex++) {
+ String vectorName =
+ vectorPrefix + generateIndexString(vectorIndex, vectorMeasurement.length);
+ measurements.add(vectorName);
+ int measurementNum = vectorMeasurement[i][vectorIndex];
+ for (int measurementIndex = 0; measurementIndex < measurementNum; measurementIndex++) {
+ String measurementName =
+ measurementPrefix + generateIndexString(measurementIndex, measurementNum);
+ measurements.add(vectorName + TsFileConstant.PATH_SEPARATOR + measurementName);
+ }
+ }
+ Collections.sort(measurements);
+ correctMeasurements.add(measurements);
+ }
+ Collections.sort(correctDevices);
+ }
+
+ /**
+ * @param devices name and number of device
+ * @param vectorMeasurement the number of device and the number of values to include in the tablet
+ * @param singleMeasurement non-vector measurement name, set null if no need
+ */
+ private void generateFile(
+ String[] devices, int[][] vectorMeasurement, String[][] singleMeasurement) {
+ try {
+ File f = FSFactoryProducer.getFSFactory().getFile(FILE_PATH);
+ if (f.exists() && !f.delete()) {
+ throw new RuntimeException("can not delete " + f.getAbsolutePath());
+ }
+ Schema schema = new Schema();
+ TsFileWriter tsFileWriter = new TsFileWriter(f, schema);
+ // write single-variable timeseries
+ if (singleMeasurement != null) {
+ for (int i = 0; i < singleMeasurement.length; i++) {
+ String device = devices[i];
+ for (String measurement : singleMeasurement[i]) {
+ tsFileWriter.registerTimeseries(
+ new Path(device, measurement),
+ new MeasurementSchema(measurement, TSDataType.INT64, TSEncoding.RLE));
+ }
+ // the number of record rows
+ int rowNum = 10;
+ for (int row = 0; row < rowNum; row++) {
+ TSRecord tsRecord = new TSRecord(row, device);
+ for (String measurement : singleMeasurement[i]) {
+ DataPoint dPoint = new LongDataPoint(measurement, row);
+ tsRecord.addTuple(dPoint);
+ }
+ if (tsRecord.dataPointList.size() > 0) {
+ tsFileWriter.write(tsRecord);
+ }
+ }
+ }
+ }
+
+ // write multi-variable timeseries
+ for (int i = 0; i < devices.length; i++) {
+ String device = devices[i];
+ logger.info("generating device {}...", device);
+ // the number of rows to include in the tablet
+ int rowNum = 10;
+ for (int vectorIndex = 0; vectorIndex < vectorMeasurement[i].length; vectorIndex++) {
+ String vectorName =
+ vectorPrefix + generateIndexString(vectorIndex, vectorMeasurement.length);
+ logger.info("generating vector {}...", vectorName);
+ List<IMeasurementSchema> measurementSchemas = new ArrayList<>();
+ int measurementNum = vectorMeasurement[i][vectorIndex];
+ String[] measurementNames = new String[measurementNum];
+ TSDataType[] dataTypes = new TSDataType[measurementNum];
+ for (int measurementIndex = 0; measurementIndex < measurementNum; measurementIndex++) {
+ String measurementName =
+ measurementPrefix + generateIndexString(measurementIndex, measurementNum);
+ logger.info("generating vector measurement {}...", measurementName);
+ // add measurements into file schema (all with INT64 data type)
+ measurementNames[measurementIndex] = measurementName;
+ dataTypes[measurementIndex] = TSDataType.INT64;
+ }
+ IMeasurementSchema measurementSchema =
+ new VectorMeasurementSchema(vectorName, measurementNames, dataTypes);
+ measurementSchemas.add(measurementSchema);
+ schema.registerTimeseries(new Path(device, vectorName), measurementSchema);
+ // add measurements into TSFileWriter
+ // construct the tablet
+ Tablet tablet = new Tablet(device, measurementSchemas);
+ long[] timestamps = tablet.timestamps;
+ Object[] values = tablet.values;
+ long timestamp = 1;
+ long value = 1000000L;
+ for (int r = 0; r < rowNum; r++, value++) {
+ int row = tablet.rowSize++;
+ timestamps[row] = timestamp++;
+ for (int j = 0; j < measurementNum; j++) {
+ long[] sensor = (long[]) values[j];
+ sensor[row] = value;
+ }
+ // write Tablet to TsFile
+ if (tablet.rowSize == tablet.getMaxRowNumber()) {
+ tsFileWriter.write(tablet);
+ tablet.reset();
+ }
+ }
+ // write Tablet to TsFile
+ if (tablet.rowSize != 0) {
+ tsFileWriter.write(tablet);
+ tablet.reset();
+ }
+ }
+ }
+ tsFileWriter.close();
+ } catch (Exception e) {
+ logger.error("meet error in TsFileWrite with tablet", e);
Review comment:
```suggestion
logger.error("meet error in TsFileWrite with tablet", e);
fail(e.getMessage());
```
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java
##########
@@ -1266,6 +1271,23 @@ public long selfCheck(
return res;
}
+ /**
+ * get metadata index node
+ *
+ * @param startOffset start read offset
+ * @param endOffset end read offset
+ * @return MetadataIndexNode
+ */
+ public MetadataIndexNode getMetadataIndexNode(long startOffset, long endOffset) {
Review comment:
Why can't we throw exceptions here?
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java
##########
@@ -647,7 +647,11 @@ private void generateMetadataIndex(
.computeIfAbsent(deviceId, k -> new ArrayList<>())
.addAll(timeseriesMetadataList);
} else {
- deviceId = metadataIndex.getName();
+ // deviceId should be determined by LEAF_DEVICE node
+ if (type.equals(MetadataIndexNodeType.LEAF_DEVICE)) {
+ deviceId = metadataIndex.getName();
+ }
+ // deviceId = metadataIndex.getName();
Review comment:
remove it.
##########
File path: session/src/test/java/org/apache/iotdb/session/IoTDBSessionVectorIT.java
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.session;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.utils.EnvironmentUtils;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.schema.IMeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.VectorMeasurementSchema;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.junit.Assert.assertEquals;
+
+/** use session interface to IT for vector timeseries insert and select Black-box Testing */
+public class IoTDBSessionVectorIT {
+ private static final String ROOT_SG1_D1_VECTOR1 = "root.sg_1.d1.vector";
+ private static final String ROOT_SG1_D1 = "root.sg_1.d1";
+ private static final String ROOT_SG1_D2 = "root.sg_1.d2";
+
+ private Session session;
+
+ @Before
+ public void setUp() throws Exception {
+ System.setProperty(IoTDBConstant.IOTDB_CONF, "src/test/resources/");
+ EnvironmentUtils.closeStatMonitor();
+ TSFileDescriptor.getInstance().getConfig().setMaxDegreeOfIndexNode(3);
+ EnvironmentUtils.envSetUp();
+ session = new Session("127.0.0.1", 6667, "root", "root");
+ session.open();
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ session.close();
+ EnvironmentUtils.cleanEnv();
+ }
+
+ @Test
+ public void alignedTest() {
+ try {
+ insertTabletWithAlignedTimeseriesMethod();
+ session.executeNonQueryStatement("flush");
+ SessionDataSet dataSet = selectTest("select * from root.sg_1.d1");
+ assertEquals(dataSet.getColumnNames().size(), 3);
+ assertEquals(dataSet.getColumnNames().get(0), "Time");
+ assertEquals(dataSet.getColumnNames().get(1), ROOT_SG1_D1_VECTOR1 + ".s1");
+ assertEquals(dataSet.getColumnNames().get(2), ROOT_SG1_D1_VECTOR1 + ".s2");
+ int time = 0;
+ while (dataSet.hasNext()) {
+ RowRecord rowRecord = dataSet.next();
+ assertEquals(rowRecord.getFields().get(0).getIntV(), time);
+ assertEquals(rowRecord.getFields().get(1).getIntV(), (time + 1) * 10 + 1);
+ assertEquals(rowRecord.getFields().get(2).getIntV(), (time + 1) * 10 + 2);
+ System.out.println(dataSet.next());
+ }
+
+ dataSet.closeOperationHandle();
+ } catch (IoTDBConnectionException e) {
+ e.printStackTrace();
Review comment:
Let the test fail if an exception is thrown.
##########
File path: tsfile/src/test/java/org/apache/iotdb/tsfile/write/MetadataIndexConstructorTest.java
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.tsfile.write;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.constant.TestConstant;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexEntry;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexNode;
+import org.apache.iotdb.tsfile.file.metadata.TimeseriesMetadata;
+import org.apache.iotdb.tsfile.file.metadata.TsFileMetadata;
+import org.apache.iotdb.tsfile.file.metadata.enums.MetadataIndexNodeType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.record.TSRecord;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.record.datapoint.DataPoint;
+import org.apache.iotdb.tsfile.write.record.datapoint.LongDataPoint;
+import org.apache.iotdb.tsfile.write.schema.IMeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.Schema;
+import org.apache.iotdb.tsfile.write.schema.VectorMeasurementSchema;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.shaded.org.apache.commons.lang.text.StrBuilder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.*;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** test for MetadataIndexConstructor */
+public class MetadataIndexConstructorTest {
+ private static final Logger logger = LoggerFactory.getLogger(MetadataIndexConstructorTest.class);
+ private final TSFileConfig conf = TSFileDescriptor.getInstance().getConfig();
+ private static final String FILE_PATH =
+ TestConstant.BASE_OUTPUT_PATH.concat("MetadataIndexConstructorTest.tsfile");
+
+ private static final String measurementPrefix = "sensor_";
+ private static final String vectorPrefix = "vector_";
+ private int maxDegreeOfIndexNode;
+
+ @Before
+ public void before() {
+ maxDegreeOfIndexNode = conf.getMaxDegreeOfIndexNode();
+ conf.setMaxDegreeOfIndexNode(10);
+ }
+
+ @After
+ public void after() {
+ conf.setMaxDegreeOfIndexNode(maxDegreeOfIndexNode);
+ }
+
+ /** Example 1: 5 entities with 5 measurements each */
+ @Test
+ public void singleIndexTest1() {
+ int deviceNum = 5;
+ int measurementNum = 5;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 2: 1 entity with 150 measurements */
+ @Test
+ public void singleIndexTest2() {
+ int deviceNum = 1;
+ int measurementNum = 150;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 3: 150 entities with 1 measurement each */
+ @Test
+ public void singleIndexTest3() {
+ int deviceNum = 150;
+ int measurementNum = 1;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + generateIndexString(i, deviceNum);
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 4: 150 entities with 150 measurements each */
+ @Test
+ public void singleIndexTest4() {
+ int deviceNum = 150;
+ int measurementNum = 1;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 5: 1 entities with 2 vectors, 9 measurements for each vector */
+ @Test
+ public void vectorIndexTest1() {
+ String[] devices = {"d0"};
+ int[][] vectorMeasurement = {{9, 9}};
+ test(devices, vectorMeasurement, null);
+ }
+
+ /** Example 6: 1 entities with 2 vectors, 15 measurements for each vector */
+ @Test
+ public void vectorIndexTest2() {
+ String[] devices = {"d0"};
+ int[][] vectorMeasurement = {{15, 15}};
+ test(devices, vectorMeasurement, null);
+ }
+
+ /**
+ * Example 7: 2 entities, measurements of entities are shown in the following table
+ *
+ * <p>d0.s0~s4 | d0.v0.(s0~s8) | d0.z0~z3 d1.s0~s14 | d1.v0.(s0~s3)
+ */
+ @Test
+ public void compositeIndexTest() {
+ String[] devices = {"d0", "d1"};
+ int[][] vectorMeasurement = {{9}, {4}};
+ String[][] singleMeasurement = {
+ {"s0", "s1", "s2", "s3", "s4", "z0", "z1", "z2", "z3"},
+ {
+ "s00", "s01", "s02", "s03", "s04", "s05", "s06", "s07", "s08", "s09", "s10", "s11", "s12",
+ "s13", "s14"
+ }
+ };
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /**
+ * start test
+ *
+ * @param devices name and number of device
+ * @param vectorMeasurement the number of device and the number of values to include in the tablet
+ * @param singleMeasurement non-vector measurement name, set null if no need
+ */
+ private void test(String[] devices, int[][] vectorMeasurement, String[][] singleMeasurement) {
+ // 1. generate file
+ generateFile(devices, vectorMeasurement, singleMeasurement);
+ // 2. read metadata from file
+ List<String> actualDevices = new ArrayList<>(); // contains all device by sequence
+ List<List<String>> actualMeasurements =
+ new ArrayList<>(); // contains all measurements group by device
+ readMetaDataDFS(actualDevices, actualMeasurements);
+ // 3. generate correct result
+ List<String> correctDevices = new ArrayList<>(); // contains all device by sequence
+ List<List<String>> correctFirstMeasurements =
+ new ArrayList<>(); // contains first measurements of every leaf, group by device
+ generateCorrectResult(
+ correctDevices, correctFirstMeasurements, devices, vectorMeasurement, singleMeasurement);
+ // 4. compare correct result with TsFile's metadata
+ Arrays.sort(devices);
+ // 4.1 make sure device in order
+ assertEquals(correctDevices.size(), devices.length);
+ assertEquals(actualDevices.size(), correctDevices.size());
+ for (int i = 0; i < actualDevices.size(); i++) {
+ assertEquals(actualDevices.get(i), correctDevices.get(i));
+ }
+ // 4.2 make sure timeseries in order
+ try (TsFileSequenceReader reader = new TsFileSequenceReader(FILE_PATH)) {
+ Map<String, List<TimeseriesMetadata>> allTimeseriesMetadata =
+ reader.getAllTimeseriesMetadata();
+ for (int j = 0; j < actualDevices.size(); j++) {
+ for (int i = 0; i < actualMeasurements.get(j).size(); i++) {
+ assertEquals(
+ allTimeseriesMetadata.get(actualDevices.get(j)).get(i).getMeasurementId(),
+ correctFirstMeasurements.get(j).get(i));
+ }
+ }
+ } catch (IOException e) {
+ e.printStackTrace();
Review comment:
```suggestion
e.printStackTrace();
fail(e.getMessage());
```
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java
##########
@@ -1175,6 +1179,7 @@ public long selfCheck(
}
truncatedSize = this.position() - 1;
} catch (Exception e) {
+ e.printStackTrace();
Review comment:
remove it.
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/MetadataIndexConstructor.java
##########
@@ -64,47 +70,22 @@ public static MetadataIndexNode constructMetadataIndex(
int serializedTimeseriesMetadataNum = 0;
for (int i = 0; i < entry.getValue().size(); i++) {
timeseriesMetadata = entry.getValue().get(i);
- if (timeseriesMetadata.isTimeColumn()) {
- // calculate the number of value columns in this vector
- int numOfValueColumns = 0;
- for (int j = i + 1; j < entry.getValue().size(); j++) {
- if (entry.getValue().get(j).isValueColumn()) {
- numOfValueColumns++;
- } else {
- break;
- }
+ if (serializedTimeseriesMetadataNum == 0
+ || serializedTimeseriesMetadataNum >= config.getMaxDegreeOfIndexNode()) {
+ if (currentIndexNode.isFull()) {
+ addCurrentIndexNodeToQueue(currentIndexNode, measurementMetadataIndexQueue, out);
+ currentIndexNode = new MetadataIndexNode(MetadataIndexNodeType.LEAF_MEASUREMENT);
}
-
- // for each vector, add time column of vector into LEAF_MEASUREMENT node
currentIndexNode.addEntry(
new MetadataIndexEntry(timeseriesMetadata.getMeasurementId(), out.getPosition()));
serializedTimeseriesMetadataNum = 0;
-
- timeseriesMetadata.serializeTo(out.wrapAsStream());
- serializedTimeseriesMetadataNum++;
- for (int j = 0; j < numOfValueColumns; j++) {
- i++;
- timeseriesMetadata = entry.getValue().get(i);
- // value columns of vector should not be added into LEAF_MEASUREMENT node
- timeseriesMetadata.serializeTo(out.wrapAsStream());
- serializedTimeseriesMetadataNum++;
- }
- } else {
- // when constructing from leaf node, every "degree number of nodes" are related to an
- // entry
- if (serializedTimeseriesMetadataNum == 0
- || serializedTimeseriesMetadataNum >= config.getMaxDegreeOfIndexNode()) {
- if (currentIndexNode.isFull()) {
- addCurrentIndexNodeToQueue(currentIndexNode, measurementMetadataIndexQueue, out);
- currentIndexNode = new MetadataIndexNode(MetadataIndexNodeType.LEAF_MEASUREMENT);
- }
- currentIndexNode.addEntry(
- new MetadataIndexEntry(timeseriesMetadata.getMeasurementId(), out.getPosition()));
- serializedTimeseriesMetadataNum = 0;
- }
- timeseriesMetadata.serializeTo(out.wrapAsStream());
- serializedTimeseriesMetadataNum++;
}
+ logger.debug(
Review comment:
remove this line and the logger.
##########
File path: tsfile/src/test/java/org/apache/iotdb/tsfile/write/MetadataIndexConstructorTest.java
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.tsfile.write;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.constant.TestConstant;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexEntry;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexNode;
+import org.apache.iotdb.tsfile.file.metadata.TimeseriesMetadata;
+import org.apache.iotdb.tsfile.file.metadata.TsFileMetadata;
+import org.apache.iotdb.tsfile.file.metadata.enums.MetadataIndexNodeType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.record.TSRecord;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.record.datapoint.DataPoint;
+import org.apache.iotdb.tsfile.write.record.datapoint.LongDataPoint;
+import org.apache.iotdb.tsfile.write.schema.IMeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.Schema;
+import org.apache.iotdb.tsfile.write.schema.VectorMeasurementSchema;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.shaded.org.apache.commons.lang.text.StrBuilder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.*;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** test for MetadataIndexConstructor */
+public class MetadataIndexConstructorTest {
+ private static final Logger logger = LoggerFactory.getLogger(MetadataIndexConstructorTest.class);
+ private final TSFileConfig conf = TSFileDescriptor.getInstance().getConfig();
+ private static final String FILE_PATH =
+ TestConstant.BASE_OUTPUT_PATH.concat("MetadataIndexConstructorTest.tsfile");
+
+ private static final String measurementPrefix = "sensor_";
+ private static final String vectorPrefix = "vector_";
+ private int maxDegreeOfIndexNode;
+
+ @Before
+ public void before() {
+ maxDegreeOfIndexNode = conf.getMaxDegreeOfIndexNode();
+ conf.setMaxDegreeOfIndexNode(10);
+ }
+
+ @After
+ public void after() {
+ conf.setMaxDegreeOfIndexNode(maxDegreeOfIndexNode);
+ }
+
+ /** Example 1: 5 entities with 5 measurements each */
+ @Test
+ public void singleIndexTest1() {
+ int deviceNum = 5;
+ int measurementNum = 5;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 2: 1 entity with 150 measurements */
+ @Test
+ public void singleIndexTest2() {
+ int deviceNum = 1;
+ int measurementNum = 150;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 3: 150 entities with 1 measurement each */
+ @Test
+ public void singleIndexTest3() {
+ int deviceNum = 150;
+ int measurementNum = 1;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + generateIndexString(i, deviceNum);
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 4: 150 entities with 150 measurements each */
+ @Test
+ public void singleIndexTest4() {
+ int deviceNum = 150;
+ int measurementNum = 1;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] singleMeasurement = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ singleMeasurement[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ singleMeasurement[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /** Example 5: 1 entities with 2 vectors, 9 measurements for each vector */
+ @Test
+ public void vectorIndexTest1() {
+ String[] devices = {"d0"};
+ int[][] vectorMeasurement = {{9, 9}};
+ test(devices, vectorMeasurement, null);
+ }
+
+ /** Example 6: 1 entities with 2 vectors, 15 measurements for each vector */
+ @Test
+ public void vectorIndexTest2() {
+ String[] devices = {"d0"};
+ int[][] vectorMeasurement = {{15, 15}};
+ test(devices, vectorMeasurement, null);
+ }
+
+ /**
+ * Example 7: 2 entities, measurements of entities are shown in the following table
+ *
+ * <p>d0.s0~s4 | d0.v0.(s0~s8) | d0.z0~z3 d1.s0~s14 | d1.v0.(s0~s3)
+ */
+ @Test
+ public void compositeIndexTest() {
+ String[] devices = {"d0", "d1"};
+ int[][] vectorMeasurement = {{9}, {4}};
+ String[][] singleMeasurement = {
+ {"s0", "s1", "s2", "s3", "s4", "z0", "z1", "z2", "z3"},
+ {
+ "s00", "s01", "s02", "s03", "s04", "s05", "s06", "s07", "s08", "s09", "s10", "s11", "s12",
+ "s13", "s14"
+ }
+ };
+ test(devices, vectorMeasurement, singleMeasurement);
+ }
+
+ /**
+ * start test
+ *
+ * @param devices name and number of device
+ * @param vectorMeasurement the number of device and the number of values to include in the tablet
+ * @param singleMeasurement non-vector measurement name, set null if no need
+ */
+ private void test(String[] devices, int[][] vectorMeasurement, String[][] singleMeasurement) {
+ // 1. generate file
+ generateFile(devices, vectorMeasurement, singleMeasurement);
+ // 2. read metadata from file
+ List<String> actualDevices = new ArrayList<>(); // contains all device by sequence
+ List<List<String>> actualMeasurements =
+ new ArrayList<>(); // contains all measurements group by device
+ readMetaDataDFS(actualDevices, actualMeasurements);
+ // 3. generate correct result
+ List<String> correctDevices = new ArrayList<>(); // contains all device by sequence
+ List<List<String>> correctFirstMeasurements =
+ new ArrayList<>(); // contains first measurements of every leaf, group by device
+ generateCorrectResult(
+ correctDevices, correctFirstMeasurements, devices, vectorMeasurement, singleMeasurement);
+ // 4. compare correct result with TsFile's metadata
+ Arrays.sort(devices);
+ // 4.1 make sure device in order
+ assertEquals(correctDevices.size(), devices.length);
+ assertEquals(actualDevices.size(), correctDevices.size());
+ for (int i = 0; i < actualDevices.size(); i++) {
+ assertEquals(actualDevices.get(i), correctDevices.get(i));
+ }
+ // 4.2 make sure timeseries in order
+ try (TsFileSequenceReader reader = new TsFileSequenceReader(FILE_PATH)) {
+ Map<String, List<TimeseriesMetadata>> allTimeseriesMetadata =
+ reader.getAllTimeseriesMetadata();
+ for (int j = 0; j < actualDevices.size(); j++) {
+ for (int i = 0; i < actualMeasurements.get(j).size(); i++) {
+ assertEquals(
+ allTimeseriesMetadata.get(actualDevices.get(j)).get(i).getMeasurementId(),
+ correctFirstMeasurements.get(j).get(i));
+ }
+ }
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ // 4.3 make sure split leaf correctly
+ for (int j = 0; j < actualDevices.size(); j++) {
+ for (int i = 0; i < actualMeasurements.get(j).size(); i++) {
+ assertEquals(
+ actualMeasurements.get(j).get(i),
+ correctFirstMeasurements.get(j).get(i * conf.getMaxDegreeOfIndexNode()));
+ }
+ }
+ }
+
+ /**
+ * read TsFile metadata, load actual message in devices and measurements
+ *
+ * @param devices load actual devices
+ * @param measurements load actual measurement(first of every leaf)
+ */
+ private void readMetaDataDFS(List<String> devices, List<List<String>> measurements) {
+ try (TsFileSequenceReader reader = new TsFileSequenceReader(FILE_PATH)) {
+ TsFileMetadata tsFileMetaData = reader.readFileMetadata();
+ MetadataIndexNode metadataIndexNode = tsFileMetaData.getMetadataIndex();
+ deviceDFS(devices, measurements, reader, metadataIndexNode);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ /** DFS in device level load actual devices */
+ private void deviceDFS(
+ List<String> devices,
+ List<List<String>> measurements,
+ TsFileSequenceReader reader,
+ MetadataIndexNode node) {
+ assertTrue(
+ node.getNodeType().equals(MetadataIndexNodeType.LEAF_DEVICE)
+ || node.getNodeType().equals(MetadataIndexNodeType.INTERNAL_DEVICE));
+ for (int i = 0; i < node.getChildren().size(); i++) {
+ MetadataIndexEntry metadataIndexEntry = node.getChildren().get(i);
+ long endOffset = node.getEndOffset();
+ if (i != node.getChildren().size() - 1) {
+ endOffset = node.getChildren().get(i + 1).getOffset();
+ }
+ MetadataIndexNode subNode =
+ reader.getMetadataIndexNode(metadataIndexEntry.getOffset(), endOffset);
+ if (node.getNodeType().equals(MetadataIndexNodeType.LEAF_DEVICE)) {
+ devices.add(metadataIndexEntry.getName());
+ measurements.add(new ArrayList<>());
+ measurementDFS(devices.size() - 1, measurements, reader, subNode);
+ } else if (node.getNodeType().equals(MetadataIndexNodeType.INTERNAL_DEVICE)) {
+ deviceDFS(devices, measurements, reader, subNode);
+ }
+ }
+ }
+ /** DFS in measurement level load actual measurements */
+ private void measurementDFS(
+ int deviceIndex,
+ List<List<String>> measurements,
+ TsFileSequenceReader reader,
+ MetadataIndexNode node) {
+ assertTrue(
+ node.getNodeType().equals(MetadataIndexNodeType.LEAF_MEASUREMENT)
+ || node.getNodeType().equals(MetadataIndexNodeType.INTERNAL_MEASUREMENT));
+ for (int i = 0; i < node.getChildren().size(); i++) {
+ MetadataIndexEntry metadataIndexEntry = node.getChildren().get(i);
+ long endOffset = node.getEndOffset();
+ if (i != node.getChildren().size() - 1) {
+ endOffset = node.getChildren().get(i + 1).getOffset();
+ }
+ if (node.getNodeType().equals(MetadataIndexNodeType.LEAF_MEASUREMENT)) {
+ // 把每个叶子节点的第一个加进来
+ measurements.get(deviceIndex).add(metadataIndexEntry.getName());
+ } else if (node.getNodeType().equals(MetadataIndexNodeType.INTERNAL_MEASUREMENT)) {
+ MetadataIndexNode subNode =
+ reader.getMetadataIndexNode(metadataIndexEntry.getOffset(), endOffset);
+ measurementDFS(deviceIndex, measurements, reader, subNode);
+ }
+ }
+ }
+
+ /**
+ * generate correct devices and measurements for test Note that if the metadata index tree is
+ * re-designed, you may need to modify this function as well.
+ *
+ * @param correctDevices output
+ * @param correctMeasurements output
+ * @param devices input
+ * @param vectorMeasurement input
+ * @param singleMeasurement input
+ */
+ private void generateCorrectResult(
+ List<String> correctDevices,
+ List<List<String>> correctMeasurements,
+ String[] devices,
+ int[][] vectorMeasurement,
+ String[][] singleMeasurement) {
+ for (int i = 0; i < devices.length; i++) {
+ String device = devices[i];
+ correctDevices.add(device);
+ // generate measurement and sort
+ List<String> measurements = new ArrayList<>();
+ // single-variable measurement
+ if (singleMeasurement != null) {
+ for (int j = 0; j < singleMeasurement[i].length; j++) {
+ measurements.add(singleMeasurement[i][j]);
+ }
+ }
+ // multi-variable measurement
+ for (int vectorIndex = 0; vectorIndex < vectorMeasurement[i].length; vectorIndex++) {
+ String vectorName =
+ vectorPrefix + generateIndexString(vectorIndex, vectorMeasurement.length);
+ measurements.add(vectorName);
+ int measurementNum = vectorMeasurement[i][vectorIndex];
+ for (int measurementIndex = 0; measurementIndex < measurementNum; measurementIndex++) {
+ String measurementName =
+ measurementPrefix + generateIndexString(measurementIndex, measurementNum);
+ measurements.add(vectorName + TsFileConstant.PATH_SEPARATOR + measurementName);
+ }
+ }
+ Collections.sort(measurements);
+ correctMeasurements.add(measurements);
+ }
+ Collections.sort(correctDevices);
+ }
+
+ /**
+ * @param devices name and number of device
+ * @param vectorMeasurement the number of device and the number of values to include in the tablet
+ * @param singleMeasurement non-vector measurement name, set null if no need
+ */
+ private void generateFile(
+ String[] devices, int[][] vectorMeasurement, String[][] singleMeasurement) {
+ try {
+ File f = FSFactoryProducer.getFSFactory().getFile(FILE_PATH);
+ if (f.exists() && !f.delete()) {
+ throw new RuntimeException("can not delete " + f.getAbsolutePath());
+ }
+ Schema schema = new Schema();
+ TsFileWriter tsFileWriter = new TsFileWriter(f, schema);
Review comment:
Use try-with-resource...
##########
File path: session/src/test/java/org/apache/iotdb/session/IoTDBSessionVectorIT.java
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.session;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.utils.EnvironmentUtils;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.schema.IMeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.VectorMeasurementSchema;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.junit.Assert.assertEquals;
+
+/** use session interface to IT for vector timeseries insert and select Black-box Testing */
+public class IoTDBSessionVectorIT {
+ private static final String ROOT_SG1_D1_VECTOR1 = "root.sg_1.d1.vector";
+ private static final String ROOT_SG1_D1 = "root.sg_1.d1";
+ private static final String ROOT_SG1_D2 = "root.sg_1.d2";
+
+ private Session session;
+
+ @Before
+ public void setUp() throws Exception {
+ System.setProperty(IoTDBConstant.IOTDB_CONF, "src/test/resources/");
+ EnvironmentUtils.closeStatMonitor();
+ TSFileDescriptor.getInstance().getConfig().setMaxDegreeOfIndexNode(3);
+ EnvironmentUtils.envSetUp();
+ session = new Session("127.0.0.1", 6667, "root", "root");
+ session.open();
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ session.close();
+ EnvironmentUtils.cleanEnv();
+ }
+
+ @Test
+ public void alignedTest() {
+ try {
+ insertTabletWithAlignedTimeseriesMethod();
+ session.executeNonQueryStatement("flush");
+ SessionDataSet dataSet = selectTest("select * from root.sg_1.d1");
+ assertEquals(dataSet.getColumnNames().size(), 3);
+ assertEquals(dataSet.getColumnNames().get(0), "Time");
+ assertEquals(dataSet.getColumnNames().get(1), ROOT_SG1_D1_VECTOR1 + ".s1");
+ assertEquals(dataSet.getColumnNames().get(2), ROOT_SG1_D1_VECTOR1 + ".s2");
+ int time = 0;
+ while (dataSet.hasNext()) {
+ RowRecord rowRecord = dataSet.next();
+ assertEquals(rowRecord.getFields().get(0).getIntV(), time);
+ assertEquals(rowRecord.getFields().get(1).getIntV(), (time + 1) * 10 + 1);
+ assertEquals(rowRecord.getFields().get(2).getIntV(), (time + 1) * 10 + 2);
+ System.out.println(dataSet.next());
+ }
+
+ dataSet.closeOperationHandle();
+ } catch (IoTDBConnectionException e) {
+ e.printStackTrace();
+ } catch (StatementExecutionException e) {
+ e.printStackTrace();
+ }
+ }
+
+ @Test
+ public void nonAlignedSingleSelectTest() {
+ try {
+ insertRecord(ROOT_SG1_D1);
+ insertTabletWithAlignedTimeseriesMethod();
+ insertRecord(ROOT_SG1_D2);
+ session.executeNonQueryStatement("flush");
+ SessionDataSet dataSet = selectTest("select * from root.sg_1.d1.vector.s2");
+ assertEquals(dataSet.getColumnNames().size(), 2);
+ assertEquals(dataSet.getColumnNames().get(0), "Time");
+ assertEquals(dataSet.getColumnNames().get(1), ROOT_SG1_D1_VECTOR1 + ".s2");
+ int time = 0;
+ while (dataSet.hasNext()) {
+ RowRecord rowRecord = dataSet.next();
+ assertEquals(rowRecord.getFields().get(0).getIntV(), time);
+ assertEquals(rowRecord.getFields().get(1).getIntV(), (time + 1) * 10 + 2);
+ System.out.println(dataSet.next());
+ }
+
+ dataSet.closeOperationHandle();
+ } catch (IoTDBConnectionException e) {
+ e.printStackTrace();
+ } catch (StatementExecutionException e) {
+ e.printStackTrace();
+ }
+ }
+
+ @Test
+ public void nonAlignedVectorSelectTest() {
+ try {
+ insertRecord(ROOT_SG1_D1);
+ insertTabletWithAlignedTimeseriesMethod();
+ insertRecord(ROOT_SG1_D2);
+ session.executeNonQueryStatement("flush");
+ SessionDataSet dataSet = selectTest("select * from root.sg_1.d1.vector");
+ assertEquals(dataSet.getColumnNames().size(), 3);
+ assertEquals(dataSet.getColumnNames().get(0), "Time");
+ assertEquals(dataSet.getColumnNames().get(1), ROOT_SG1_D1_VECTOR1 + ".s1");
+ assertEquals(dataSet.getColumnNames().get(2), ROOT_SG1_D1_VECTOR1 + ".s2");
+ int time = 0;
+ while (dataSet.hasNext()) {
+ RowRecord rowRecord = dataSet.next();
+ assertEquals(rowRecord.getFields().get(0).getIntV(), time);
+ assertEquals(rowRecord.getFields().get(1).getIntV(), (time + 1) * 10 + 1);
+ assertEquals(rowRecord.getFields().get(2).getIntV(), (time + 1) * 10 + 2);
+ System.out.println(dataSet.next());
+ }
+
+ dataSet.closeOperationHandle();
+ } catch (IoTDBConnectionException e) {
+ e.printStackTrace();
+ } catch (StatementExecutionException e) {
+ e.printStackTrace();
+ }
+ }
+
+ private SessionDataSet selectTest(String sql)
+ throws StatementExecutionException, IoTDBConnectionException {
+ SessionDataSet dataSet = session.executeQueryStatement(sql);
+ System.out.println(dataSet.getColumnNames());
+ while (dataSet.hasNext()) {
+ System.out.println(dataSet.next());
+ }
+ return dataSet;
+ }
+
+ /** Method 1 for insert tablet with aligned timeseries */
+ private void insertTabletWithAlignedTimeseriesMethod()
+ throws IoTDBConnectionException, StatementExecutionException {
+ // The schema of measurements of one device
+ // only measurementId and data type in MeasurementSchema take effects in Tablet
+ List<IMeasurementSchema> schemaList = new ArrayList<>();
+ schemaList.add(
+ new VectorMeasurementSchema(
+ "vector",
+ new String[] {"s1", "s2"},
+ new TSDataType[] {TSDataType.INT64, TSDataType.INT32}));
+
+ Tablet tablet = new Tablet(ROOT_SG1_D1_VECTOR1, schemaList);
+ tablet.setAligned(true);
+ long timestamp = 0;
+
+ for (long row = 0; row < 100; row++) {
+ int rowIndex = tablet.rowSize++;
+ tablet.addTimestamp(rowIndex, timestamp);
+ tablet.addValue(
+ schemaList.get(0).getValueMeasurementIdList().get(0), rowIndex, row * 10 + 1L);
+ // new SecureRandom().nextLong());
Review comment:
Remove it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677277111
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java
##########
@@ -1266,6 +1276,23 @@ public long selfCheck(
return res;
}
+ /**
Review comment:
remove to sketchTool
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r678864948
##########
File path: docs/zh/SystemDesign/TsFile/Format.md
##########
@@ -27,39 +27,63 @@
### 1.1 变量的存储
-- **大端存储**
- - 比如： `int` `0x8` 将会被存储为 `00 00 00 08`, 而不是 `08 00 00 00`
-- **可变长的字符串类型**
- - 存储的方式是以一个 `int` 类型的 `Size` + 字符串组成。`Size` 的值可以为 0。
- - `Size` 指的是字符串所占的字节数，它并不一定等于字符串的长度。
- - 举例来说，"sensor_1" 这个字符串将被存储为 `00 00 00 08` + "sensor_1" (ASCII 编码）。
- - 另外需要注意的一点是文件签名 "TsFile000001" (`Magic String` + `Version`), 因为他的 `Size(12)` 和 ASCII 编码值是固定的，所以没有必要在这个字符串前的写入 `Size` 值。
- **数据类型**
+
- 0: BOOLEAN
- 1: INT32 (`int`)
- 2: INT64 (`long`)
- 3: FLOAT
- 4: DOUBLE
- 5: TEXT (`String`)
+
- **编码类型**
- - 0: PLAIN
- - 1: DICTIONARY
- - 2: RLE
- - 3: DIFF
- - 4: TS_2DIFF
- - 5: BITMAP
- - 6: GORILLA_V1
- - 7: REGULAR
- - 8: GORILLA
-- **压缩类型**
- - 0: UNCOMPRESSED
- - 1: SNAPPY
- - 2: GZIP
- - 3: LZO
- - 4: SDT
- - 5: PAA
- - 6: PLA
- - 7: LZ4
+
+ 为了提高数据的存储效率，需要在数据写入的过程中对数据进行编码，从而减少磁盘空间的使用量。在写数据以及读数据的过程中都能够减少I/O操作的数据量从而提高性能。IoTDB支持多种针对不同类型的数据的编码方法：
+
+ - **0: PLAIN**
+
+ - PLAIN编码，默认的编码方式，即不编码，支持多种数据类型，压缩和解压缩的时间效率较高，但空间存储效率较低。
+
+ - **1: DICTIONARY**
+
+ - 字典编码是一种无损编码。它适合编码基数小的数据（即数据去重后唯一值数量小）。不推荐用于基数大的数据。
+
+ - **2: RLE**
+
+ - 游程编码，比较适合存储某些整数值连续出现的序列，不适合编码大部分情况下前后值不一样的序列数据。
+
+ - 游程编码也可用于对浮点数进行编码，但在创建时间序列的时候需指定保留小数位数（MAX_POINT_NUMBER，具体指定方式参见本文[SQL 参考文档](https://iotdb.apache.org/zh/UserGuide/Master/Operation Manual/SQL Reference.html)）。比较适合存储某些浮点数值连续出现的序列数据，不适合存储对小数点后精度要求较高以及前后波动较大的序列数据。
Review comment:
Fix the dead link :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677268461
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/write/chunk/ChunkGroupWriterImpl.java
##########
@@ -79,10 +87,72 @@ public void write(Tablet tablet) throws WriteProcessException {
if (!chunkWriters.containsKey(measurementId)) {
throw new NoMeasurementException("measurement id" + measurementId + " not found!");
}
- writeByDataType(tablet, measurementId, dataType, i);
+ if (dataType.equals(TSDataType.VECTOR)) {
+ int valuesColumnNum = writeVectorDataType(tablet, measurementId, i);
+ i += (valuesColumnNum - 1);
+ } else {
+ writeByDataType(tablet, measurementId, dataType, i);
+ }
+ }
+ }
+
+ /**
+ * write if data type is VECTOR this method write next n column values (belong to one vector), and
+ * return n to increase index
+ *
+ * @param tablet table
+ * @param measurement vector measurement
+ * @param index measurement start index
+ * @return number of values' column
+ */
+ private int writeVectorDataType(Tablet tablet, String measurement, int index) {
+ // reference: MemTableFlushTask.java
+ int batchSize = tablet.rowSize;
+ VectorMeasurementSchema vectorMeasurementSchema =
+ (VectorMeasurementSchema) tablet.getSchemas().get(index);
+ List<TSDataType> valueDataTypes = vectorMeasurementSchema.getValueTSDataTypeList();
+ IChunkWriter vectorChunkWriter = chunkWriters.get(measurement);
+ for (int row = 0; row < batchSize; row++) {
+ long time = tablet.timestamps[row];
+ for (int columnIndex = 0; columnIndex < valueDataTypes.size(); columnIndex++) {
+ boolean isNull = false; // TODO: 这是干嘛用的？
Review comment:
delete chinese
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677275524
##########
File path: tsfile/src/test/java/org/apache/iotdb/tsfile/write/writer/VectorChunkWriterImplTest.java
##########
@@ -72,7 +74,7 @@ public void testWrite1() {
// value chunk 1
assertEquals(0x40 | MetaMarker.ONLY_ONE_PAGE_CHUNK_HEADER, ReadWriteIOUtils.readByte(buffer));
- assertEquals("s1", ReadWriteIOUtils.readVarIntString(buffer));
+ assertEquals("s1.time.s1", ReadWriteIOUtils.readVarIntString(buffer));
Review comment:
rename -> vectorName
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677246218
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/write/chunk/TimeChunkWriter.java
##########
@@ -235,6 +235,7 @@ public void writeAllPagesOfChunkToTsFile(TsFileIOWriter writer) throws IOExcepti
// write all pages of this column
writer.writeBytesToStream(pageBuffer);
+ logger.debug("write time chunk {} from {} to {}", measurementId, dataOffset, writer.getPos());
Review comment:
Remove this debug log.
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/write/chunk/ValueChunkWriter.java
##########
@@ -234,6 +234,7 @@ public void writeAllPagesOfChunkToTsFile(TsFileIOWriter writer) throws IOExcepti
writer.writeBytesToStream(pageBuffer);
int dataSize = (int) (writer.getPos() - dataOffset);
+ logger.debug("write value chunk {} from {} to {}", measurementId, dataOffset, writer.getPos());
Review comment:
Remove this debug log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677264280
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/MetadataIndexConstructor.java
##########
@@ -120,28 +122,31 @@ public static MetadataIndexNode constructMetadataIndex(
new MetadataIndexNode(MetadataIndexNodeType.LEAF_DEVICE);
for (Map.Entry<String, MetadataIndexNode> entry : deviceMetadataIndexMap.entrySet()) {
metadataIndexNode.addEntry(new MetadataIndexEntry(entry.getKey(), out.getPosition()));
+ logger.debug(
Review comment:
delete
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/MetadataIndexConstructor.java
##########
@@ -171,6 +176,8 @@ private static MetadataIndexNode generateRootNode(
}
currentIndexNode.addEntry(
new MetadataIndexEntry(metadataIndexNode.peek().getName(), out.getPosition()));
+
Review comment:
delete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#issuecomment-887299845
[](https://coveralls.io/builds/41674981)
Coverage increased (+0.009%) to 68.078% when pulling **c6caf22838b3d46d29a7ce81c9a654780cf463f8 on Cpaulyz:TsFileTailIndex** into **47fa594403d5bb0c79d8414886b5a9fe6a24b778 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677277869
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java
##########
@@ -258,6 +258,12 @@ public byte readVersionNumber() throws IOException {
public TsFileMetadata readFileMetadata() throws IOException {
try {
if (tsFileMetaData == null) {
+
Review comment:
delete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#issuecomment-887299845
[](https://coveralls.io/builds/41675388)
Coverage decreased (-0.001%) to 68.068% when pulling **c6caf22838b3d46d29a7ce81c9a654780cf463f8 on Cpaulyz:TsFileTailIndex** into **47fa594403d5bb0c79d8414886b5a9fe6a24b778 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677270567
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/write/chunk/ChunkWriterImpl.java
##########
@@ -440,6 +440,12 @@ private void writeAllPagesOfChunkToTsFile(TsFileIOWriter writer, Statistics<?> s
writer.writeBytesToStream(pageBuffer);
int dataSize = (int) (writer.getPos() - dataOffset);
+ logger.debug(
Review comment:
delete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#issuecomment-887299845
[](https://coveralls.io/builds/41703876)
Coverage decreased (-0.03%) to 68.183% when pulling **f5a20930f43c5603139f7b13c63ac155534033c3 on Cpaulyz:TsFileTailIndex** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] wangchao316 commented on a change in pull request #3627: [IOTDB-1517] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
wangchao316 commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r676448942
##########
File path: tsfile/src/test/java/org/apache/iotdb/tsfile/write/MetadataIndexConstructorTest.java
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.tsfile.write;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.constant.TestConstant;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexEntry;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexNode;
+import org.apache.iotdb.tsfile.file.metadata.TimeseriesMetadata;
+import org.apache.iotdb.tsfile.file.metadata.TsFileMetadata;
+import org.apache.iotdb.tsfile.file.metadata.enums.MetadataIndexNodeType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.record.TSRecord;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.record.datapoint.DataPoint;
+import org.apache.iotdb.tsfile.write.record.datapoint.LongDataPoint;
+import org.apache.iotdb.tsfile.write.schema.IMeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.Schema;
+import org.apache.iotdb.tsfile.write.schema.VectorMeasurementSchema;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.shaded.org.apache.commons.lang.text.StrBuilder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.*;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** test for MetadataIndexConstructor */
+public class MetadataIndexConstructorTest {
+ private static final Logger logger = LoggerFactory.getLogger(MetadataIndexConstructorTest.class);
+ private final TSFileConfig conf = TSFileDescriptor.getInstance().getConfig();
+ private static final String FILE_PATH =
+ TestConstant.BASE_OUTPUT_PATH.concat("MetadataIndexConstructorTest.tsfile");
+
+ private static final String measurementPrefix = "sensor_";
+ private static final String vectorPrefix = "vector_";
+ private int maxDegreeOfIndexNode;
+
+ @Before
+ public void before() {
+ maxDegreeOfIndexNode = conf.getMaxDegreeOfIndexNode();
+ conf.setMaxDegreeOfIndexNode(10);
+ }
+
+ @After
+ public void after() {
+ conf.setMaxDegreeOfIndexNode(maxDegreeOfIndexNode);
+ }
+
+ /** 5个实体，每个实体有5个物理量（docs.例1） */
+ @Test
+ public void singleIndexTest1() {
+ int deviceNum = 5;
+ int measurementNum = 5;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] measurements = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ measurements[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ measurements[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, measurements);
+ }
+
+ /** 1个实体，每个实体有150个物理量（docs.例2） */
+ @Test
+ public void singleIndexTest2() {
+ int deviceNum = 1;
+ int measurementNum = 150;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] measurements = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ measurements[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ measurements[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, measurements);
+ }
+
+ /** 150个实体，每个实体有1个物理量（docs.例3） */
Review comment:
the same
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/MetadataIndexConstructor.java
##########
@@ -62,49 +68,45 @@ public static MetadataIndexNode constructMetadataIndex(
MetadataIndexNode currentIndexNode =
new MetadataIndexNode(MetadataIndexNodeType.LEAF_MEASUREMENT);
int serializedTimeseriesMetadataNum = 0;
+ String vectorName = ""; // record previous vector name
+ int numOfValueColumns = 0;
for (int i = 0; i < entry.getValue().size(); i++) {
timeseriesMetadata = entry.getValue().get(i);
- if (timeseriesMetadata.isTimeColumn()) {
- // calculate the number of value columns in this vector
- int numOfValueColumns = 0;
+ if (numOfValueColumns > 0) {
+ // must be value column，不清楚这里是否需要加一层检验？外层可否保证？
Review comment:
Good job.Thanks for your contribution.
I have a little question.
generate , chinese is used in notes
##########
File path: tsfile/src/test/java/org/apache/iotdb/tsfile/write/MetadataIndexConstructorTest.java
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.tsfile.write;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.constant.TestConstant;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexEntry;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexNode;
+import org.apache.iotdb.tsfile.file.metadata.TimeseriesMetadata;
+import org.apache.iotdb.tsfile.file.metadata.TsFileMetadata;
+import org.apache.iotdb.tsfile.file.metadata.enums.MetadataIndexNodeType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.record.TSRecord;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.record.datapoint.DataPoint;
+import org.apache.iotdb.tsfile.write.record.datapoint.LongDataPoint;
+import org.apache.iotdb.tsfile.write.schema.IMeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.Schema;
+import org.apache.iotdb.tsfile.write.schema.VectorMeasurementSchema;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.shaded.org.apache.commons.lang.text.StrBuilder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.*;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** test for MetadataIndexConstructor */
+public class MetadataIndexConstructorTest {
+ private static final Logger logger = LoggerFactory.getLogger(MetadataIndexConstructorTest.class);
+ private final TSFileConfig conf = TSFileDescriptor.getInstance().getConfig();
+ private static final String FILE_PATH =
+ TestConstant.BASE_OUTPUT_PATH.concat("MetadataIndexConstructorTest.tsfile");
+
+ private static final String measurementPrefix = "sensor_";
+ private static final String vectorPrefix = "vector_";
+ private int maxDegreeOfIndexNode;
+
+ @Before
+ public void before() {
+ maxDegreeOfIndexNode = conf.getMaxDegreeOfIndexNode();
+ conf.setMaxDegreeOfIndexNode(10);
+ }
+
+ @After
+ public void after() {
+ conf.setMaxDegreeOfIndexNode(maxDegreeOfIndexNode);
+ }
+
+ /** 5个实体，每个实体有5个物理量（docs.例1） */
+ @Test
+ public void singleIndexTest1() {
+ int deviceNum = 5;
+ int measurementNum = 5;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] measurements = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ measurements[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ measurements[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, measurements);
+ }
+
+ /** 1个实体，每个实体有150个物理量（docs.例2） */
+ @Test
+ public void singleIndexTest2() {
+ int deviceNum = 1;
+ int measurementNum = 150;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] measurements = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ measurements[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ measurements[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, measurements);
+ }
+
+ /** 150个实体，每个实体有1个物理量（docs.例3） */
+ @Test
+ public void singleIndexTest3() {
+ int deviceNum = 150;
+ int measurementNum = 1;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] measurements = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + generateIndexString(i, deviceNum);
+ vectorMeasurement[i] = new int[0];
+ measurements[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ measurements[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, measurements);
+ }
+
+ /** 150个实体，每个实体有150个物理量（docs.例4） */
+ @Test
+ public void singleIndexTest4() {
+ int deviceNum = 150;
+ int measurementNum = 1;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] measurements = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ measurements[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ measurements[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, measurements);
+ }
+
+ /** 1个实体，20个物理量，2个多元时间序列组，每个多元时间序列组分别有9个物理量（docs.例5） */
+ @Test
+ public void vectorIndexTest1() {
+ String[] devices = {"d0"};
+ int[][] vectorMeasurement = {{9, 9}};
+ test(devices, vectorMeasurement, null);
+ }
+
+ /** 1个实体，30个物理量，2个多元时间序列组，每个多元时间序列组分别有15个物理量（docs.例6） */
+ @Test
+ public void vectorIndexTest2() {
+ String[] devices = {"d0"};
+ int[][] vectorMeasurement = {{15, 15}};
+ test(devices, vectorMeasurement, null);
+ }
+
+ /**
+ * 综合测试，包括多元时间序列与简单时间序列（docs.例7）
+ *
+ * <p>d0.s0~s4 | d0.v0.(s0~s8) | d0.z0~z3 d1.s0~s14 | d1.v0.(s0~s3)
+ */
+ @Test
+ public void compositeIndexTest() {
+ String[] devices = {"d0", "d1"};
+ int[][] vectorMeasurement = {{9}, {4}};
+ String[][] measurements = {
+ {"s0", "s1", "s2", "s3", "s4", "z0", "z1", "z2", "z3"},
+ {
+ "s00", "s01", "s02", "s03", "s04", "s05", "s06", "s07", "s08", "s09", "s10", "s11", "s12",
+ "s13", "s14"
+ }
+ };
+ test(devices, vectorMeasurement, measurements);
+ }
+
+ /**
+ * start test
+ *
+ * @param devices name and number of device
+ * @param vectorMeasurement the number of device and the number of values to include in the tablet
+ * @param singleMeasurement non-vector measurement name, set null if no need
+ */
+ private void test(String[] devices, int[][] vectorMeasurement, String[][] singleMeasurement) {
+ // 1. generate file
+ generateFile(devices, vectorMeasurement, singleMeasurement);
+ // 2. read metadata from file
+ List<String> actualDevices = new ArrayList<>(); // contains all device by sequence
+ List<List<String>> actualMeasurements =
+ new ArrayList<>(); // contains all measurements group by device
+ readMetaDataDFS(actualDevices, actualMeasurements);
+ // 3. generate correct result
+ List<String> correctDevices = new ArrayList<>(); // contains all device by sequence
+ List<List<String>> correctFirstMeasurements =
+ new ArrayList<>(); // contains first measurements of every leaf, group by device
+ generateCorrectResult(
+ correctDevices, correctFirstMeasurements, devices, vectorMeasurement, singleMeasurement);
+ // 4. compare correct result with TsFile's metadata
+ Arrays.sort(devices);
+ // 4.1 make sure device in order
+ assertEquals(correctDevices.size(), devices.length);
+ assertEquals(actualDevices.size(), correctDevices.size());
+ for (int i = 0; i < actualDevices.size(); i++) {
+ assertEquals(actualDevices.get(i), correctDevices.get(i));
+ }
+ // 4.2 make sure timeseries in order
+ try (TsFileSequenceReader reader = new TsFileSequenceReader(FILE_PATH)) {
+ Map<String, List<TimeseriesMetadata>> allTimeseriesMetadata =
+ reader.getAllTimeseriesMetadata();
+ for (int j = 0; j < actualDevices.size(); j++) {
+ for (int i = 0; i < actualMeasurements.get(j).size(); i++) {
+ assertEquals(
+ allTimeseriesMetadata.get(actualDevices.get(j)).get(i).getMeasurementId(),
+ correctFirstMeasurements.get(j).get(i));
+ }
+ }
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ // 4.3 make sure split leaf correctly
+ for (int j = 0; j < actualDevices.size(); j++) {
+ for (int i = 0; i < actualMeasurements.get(j).size(); i++) {
+ assertEquals(
+ actualMeasurements.get(j).get(i),
+ correctFirstMeasurements.get(j).get(i * conf.getMaxDegreeOfIndexNode()));
+ }
+ }
+ }
+
+ /**
+ * read TsFile metadata, load actual message in devices and measurements
+ *
+ * @param devices load actual devices
+ * @param measurements load actual measurement(first of every leaf)
+ */
+ private void readMetaDataDFS(List<String> devices, List<List<String>> measurements) {
+ try (TsFileSequenceReader reader = new TsFileSequenceReader(FILE_PATH)) {
+ TsFileMetadata tsFileMetaData = reader.readFileMetadata();
+ MetadataIndexNode metadataIndexNode = tsFileMetaData.getMetadataIndex();
+ deviceDFS(devices, measurements, reader, metadataIndexNode);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ /** DFS in device level load actual devices */
+ private void deviceDFS(
+ List<String> devices,
+ List<List<String>> measurements,
+ TsFileSequenceReader reader,
+ MetadataIndexNode node) {
+ assertTrue(
+ node.getNodeType().equals(MetadataIndexNodeType.LEAF_DEVICE)
+ || node.getNodeType().equals(MetadataIndexNodeType.INTERNAL_DEVICE));
+ for (int i = 0; i < node.getChildren().size(); i++) {
+ MetadataIndexEntry metadataIndexEntry = node.getChildren().get(i);
+ long endOffset = node.getEndOffset();
+ if (i != node.getChildren().size() - 1) {
+ endOffset = node.getChildren().get(i + 1).getOffset();
+ }
+ MetadataIndexNode subNode =
+ reader.getMetadataIndexNode(metadataIndexEntry.getOffset(), endOffset);
+ if (node.getNodeType().equals(MetadataIndexNodeType.LEAF_DEVICE)) {
+ devices.add(metadataIndexEntry.getName());
+ measurements.add(new ArrayList<>());
+ measurementDFS(devices.size() - 1, measurements, reader, subNode);
+ } else if (node.getNodeType().equals(MetadataIndexNodeType.INTERNAL_DEVICE)) {
+ deviceDFS(devices, measurements, reader, subNode);
+ }
+ }
+ }
+ /** DFS in measurement level load actual measurements */
+ private void measurementDFS(
+ int deviceIndex,
+ List<List<String>> measurements,
+ TsFileSequenceReader reader,
+ MetadataIndexNode node) {
+ assertTrue(
+ node.getNodeType().equals(MetadataIndexNodeType.LEAF_MEASUREMENT)
+ || node.getNodeType().equals(MetadataIndexNodeType.INTERNAL_MEASUREMENT));
+ for (int i = 0; i < node.getChildren().size(); i++) {
+ MetadataIndexEntry metadataIndexEntry = node.getChildren().get(i);
+ long endOffset = node.getEndOffset();
+ if (i != node.getChildren().size() - 1) {
+ endOffset = node.getChildren().get(i + 1).getOffset();
+ }
+ if (node.getNodeType().equals(MetadataIndexNodeType.LEAF_MEASUREMENT)) {
+ // 把每个叶子节点的第一个加进来
Review comment:
the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#issuecomment-887299845
[](https://coveralls.io/builds/41682273)
Coverage decreased (-0.02%) to 68.185% when pulling **947eb94f2f59681be000db2f4be03f1788f1b161 on Cpaulyz:TsFileTailIndex** into **716710c4466e4c9f4eafc8edb38cf8b6ac489c40 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#issuecomment-887299845
[](https://coveralls.io/builds/41742263)
Coverage decreased (-0.0001%) to 68.213% when pulling **91bec14b656435d22ccd858b1e13c8b5fb055012 on Cpaulyz:TsFileTailIndex** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#issuecomment-887299845
[](https://coveralls.io/builds/41710979)
Coverage decreased (-0.03%) to 68.181% when pulling **e3165d8d1b23c6ec51f88d5c358a1dcdbec40c74 on Cpaulyz:TsFileTailIndex** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#issuecomment-887259330
rename TsFileSketchTool
* `ChunkMetadataList` -> `TimeseriesIndex`
* `ChunkMetadata` -> `ChunkIndex`
update document and fix some document bug, combine PR #3613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677270892
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/write/writer/TsFileIOWriter.java
##########
@@ -354,6 +364,10 @@ private void flushOneChunkMetadata(
if (!chunkMetadata.getDataType().equals(dataType)) {
continue;
}
+ logger.debug(
Review comment:
remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu merged pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
SteveYurongSu merged pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#issuecomment-887299845
[](https://coveralls.io/builds/41674605)
Coverage increased (+0.02%) to 68.086% when pulling **c6caf22838b3d46d29a7ce81c9a654780cf463f8 on Cpaulyz:TsFileTailIndex** into **47fa594403d5bb0c79d8414886b5a9fe6a24b778 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677263387
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/MetadataIndexConstructor.java
##########
@@ -62,49 +68,45 @@ public static MetadataIndexNode constructMetadataIndex(
MetadataIndexNode currentIndexNode =
new MetadataIndexNode(MetadataIndexNodeType.LEAF_MEASUREMENT);
int serializedTimeseriesMetadataNum = 0;
+ String vectorName = ""; // record previous vector name
+ int numOfValueColumns = 0;
for (int i = 0; i < entry.getValue().size(); i++) {
timeseriesMetadata = entry.getValue().get(i);
- if (timeseriesMetadata.isTimeColumn()) {
- // calculate the number of value columns in this vector
- int numOfValueColumns = 0;
+ if (numOfValueColumns > 0) {
+ // must be value column
+ numOfValueColumns--;
+ timeseriesMetadata.setMeasurementId(timeseriesMetadata.getMeasurementId());
Review comment:
useless
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677243896
##########
File path: docs/zh/SystemDesign/TsFile/Format.md
##########
@@ -21,46 +21,74 @@
# TsFile 文件格式
+[TOC]
Review comment:
Remove it~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#issuecomment-887299845
[](https://coveralls.io/builds/41682352)
Coverage decreased (-0.04%) to 68.16% when pulling **947eb94f2f59681be000db2f4be03f1788f1b161 on Cpaulyz:TsFileTailIndex** into **716710c4466e4c9f4eafc8edb38cf8b6ac489c40 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#issuecomment-887299845
[](https://coveralls.io/builds/41741786)
Coverage decreased (-0.008%) to 68.205% when pulling **91bec14b656435d22ccd858b1e13c8b5fb055012 on Cpaulyz:TsFileTailIndex** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677253253
##########
File path: tsfile/src/test/java/org/apache/iotdb/tsfile/write/MetadataIndexConstructorTest.java
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.tsfile.write;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.constant.TestConstant;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexEntry;
+import org.apache.iotdb.tsfile.file.metadata.MetadataIndexNode;
+import org.apache.iotdb.tsfile.file.metadata.TimeseriesMetadata;
+import org.apache.iotdb.tsfile.file.metadata.TsFileMetadata;
+import org.apache.iotdb.tsfile.file.metadata.enums.MetadataIndexNodeType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.record.TSRecord;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.record.datapoint.DataPoint;
+import org.apache.iotdb.tsfile.write.record.datapoint.LongDataPoint;
+import org.apache.iotdb.tsfile.write.schema.IMeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.Schema;
+import org.apache.iotdb.tsfile.write.schema.VectorMeasurementSchema;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.shaded.org.apache.commons.lang.text.StrBuilder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.*;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** test for MetadataIndexConstructor */
+public class MetadataIndexConstructorTest {
+ private static final Logger logger = LoggerFactory.getLogger(MetadataIndexConstructorTest.class);
+ private final TSFileConfig conf = TSFileDescriptor.getInstance().getConfig();
+ private static final String FILE_PATH =
+ TestConstant.BASE_OUTPUT_PATH.concat("MetadataIndexConstructorTest.tsfile");
+
+ private static final String measurementPrefix = "sensor_";
+ private static final String vectorPrefix = "vector_";
+ private int maxDegreeOfIndexNode;
+
+ @Before
+ public void before() {
+ maxDegreeOfIndexNode = conf.getMaxDegreeOfIndexNode();
+ conf.setMaxDegreeOfIndexNode(10);
+ }
+
+ @After
+ public void after() {
+ conf.setMaxDegreeOfIndexNode(maxDegreeOfIndexNode);
+ }
+
+ /** Example 1: 5 entities with 5 measurements each */
+ @Test
+ public void singleIndexTest1() {
+ int deviceNum = 5;
+ int measurementNum = 5;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] measurements = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ measurements[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ measurements[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, measurements);
+ }
+
+ /** Example 2: 1 entity with 150 measurements */
+ @Test
+ public void singleIndexTest2() {
+ int deviceNum = 1;
+ int measurementNum = 150;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] measurements = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ measurements[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ measurements[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, measurements);
+ }
+
+ /** Example 3: 150 entities with 1 measurement each */
+ @Test
+ public void singleIndexTest3() {
+ int deviceNum = 150;
+ int measurementNum = 1;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] measurements = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + generateIndexString(i, deviceNum);
+ vectorMeasurement[i] = new int[0];
+ measurements[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ measurements[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, measurements);
+ }
+
+ /** Example 4: 150 entities with 150 measurements each */
+ @Test
+ public void singleIndexTest4() {
+ int deviceNum = 150;
+ int measurementNum = 1;
+ String[] devices = new String[deviceNum];
+ int[][] vectorMeasurement = new int[deviceNum][];
+ String[][] measurements = new String[deviceNum][];
+ for (int i = 0; i < deviceNum; i++) {
+ devices[i] = "d" + i;
+ vectorMeasurement[i] = new int[0];
+ measurements[i] = new String[measurementNum];
+ for (int j = 0; j < measurementNum; j++) {
+ measurements[i][j] = measurementPrefix + generateIndexString(j, measurementNum);
+ }
+ }
+ test(devices, vectorMeasurement, measurements);
+ }
+
+ /** Example 5: 1 entities with 2 vectors, 9 measurements for each vector */
+ @Test
+ public void vectorIndexTest1() {
+ String[] devices = {"d0"};
+ int[][] vectorMeasurement = {{9, 9}};
+ test(devices, vectorMeasurement, null);
+ }
+
+ /** Example 6: 1 entities with 2 vectors, 15 measurements for each vector */
+ @Test
+ public void vectorIndexTest2() {
+ String[] devices = {"d0"};
+ int[][] vectorMeasurement = {{15, 15}};
+ test(devices, vectorMeasurement, null);
+ }
+
+ /**
+ * Example 7: 2 entities, measurements of entities are shown in the following table
+ *
+ * <p>d0.s0~s4 | d0.v0.(s0~s8) | d0.z0~z3 d1.s0~s14 | d1.v0.(s0~s3)
+ */
+ @Test
+ public void compositeIndexTest() {
+ String[] devices = {"d0", "d1"};
+ int[][] vectorMeasurement = {{9}, {4}};
+ String[][] measurements = {
Review comment:
Rename it to singleMeasurement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677270117
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/write/chunk/ChunkGroupWriterImpl.java
##########
@@ -79,10 +87,72 @@ public void write(Tablet tablet) throws WriteProcessException {
if (!chunkWriters.containsKey(measurementId)) {
throw new NoMeasurementException("measurement id" + measurementId + " not found!");
}
- writeByDataType(tablet, measurementId, dataType, i);
+ if (dataType.equals(TSDataType.VECTOR)) {
+ int valuesColumnNum = writeVectorDataType(tablet, measurementId, i);
+ i += (valuesColumnNum - 1);
+ } else {
+ writeByDataType(tablet, measurementId, dataType, i);
+ }
+ }
+ }
+
+ /**
+ * write if data type is VECTOR this method write next n column values (belong to one vector), and
+ * return n to increase index
+ *
+ * @param tablet table
+ * @param measurement vector measurement
+ * @param index measurement start index
+ * @return number of values' column
+ */
+ private int writeVectorDataType(Tablet tablet, String measurement, int index) {
+ // reference: MemTableFlushTask.java
+ int batchSize = tablet.rowSize;
+ VectorMeasurementSchema vectorMeasurementSchema =
+ (VectorMeasurementSchema) tablet.getSchemas().get(index);
+ List<TSDataType> valueDataTypes = vectorMeasurementSchema.getValueTSDataTypeList();
+ IChunkWriter vectorChunkWriter = chunkWriters.get(measurement);
+ for (int row = 0; row < batchSize; row++) {
+ long time = tablet.timestamps[row];
+ for (int columnIndex = 0; columnIndex < valueDataTypes.size(); columnIndex++) {
+ boolean isNull = false; // TODO: 这是干嘛用的？
Review comment:
isnull
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677263923
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/MetadataIndexConstructor.java
##########
@@ -62,49 +68,45 @@ public static MetadataIndexNode constructMetadataIndex(
MetadataIndexNode currentIndexNode =
new MetadataIndexNode(MetadataIndexNodeType.LEAF_MEASUREMENT);
int serializedTimeseriesMetadataNum = 0;
+ String vectorName = ""; // record previous vector name
+ int numOfValueColumns = 0;
for (int i = 0; i < entry.getValue().size(); i++) {
timeseriesMetadata = entry.getValue().get(i);
- if (timeseriesMetadata.isTimeColumn()) {
- // calculate the number of value columns in this vector
- int numOfValueColumns = 0;
+ if (numOfValueColumns > 0) {
+ // must be value column
+ numOfValueColumns--;
+ timeseriesMetadata.setMeasurementId(timeseriesMetadata.getMeasurementId());
+ } else if (timeseriesMetadata.isTimeColumn()) {
+ vectorName = timeseriesMetadata.getMeasurementId();
for (int j = i + 1; j < entry.getValue().size(); j++) {
if (entry.getValue().get(j).isValueColumn()) {
numOfValueColumns++;
} else {
break;
}
}
+ }
+ if (serializedTimeseriesMetadataNum == 0
+ || serializedTimeseriesMetadataNum >= config.getMaxDegreeOfIndexNode()) {
+ if (currentIndexNode.isFull()) {
+ addCurrentIndexNodeToQueue(currentIndexNode, measurementMetadataIndexQueue, out);
+ currentIndexNode = new MetadataIndexNode(MetadataIndexNodeType.LEAF_MEASUREMENT);
+ }
- // for each vector, add time column of vector into LEAF_MEASUREMENT node
+ logger.debug(
Review comment:
delete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677282145
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java
##########
@@ -647,7 +653,11 @@ private void generateMetadataIndex(
.computeIfAbsent(deviceId, k -> new ArrayList<>())
.addAll(timeseriesMetadataList);
} else {
- deviceId = metadataIndex.getName();
+ // deviceId should be determined by LEAF_DEVICE node
+ if (type.equals(MetadataIndexNodeType.LEAF_DEVICE)) {
Review comment:
check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677299107
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/cache/TimeSeriesMetadataCache.java
##########
@@ -251,6 +246,9 @@ public TimeseriesMetadata get(
boolean debug)
throws IOException {
// put all sub sensors into allSensors
+ for (int i = 0; i < subSensorList.size(); i++) {
Review comment:
allSensors.removeAll(subSensorList); ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls commented on pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#issuecomment-887299845
[](https://coveralls.io/builds/41674006)
Coverage decreased (-0.01%) to 68.057% when pulling **c6caf22838b3d46d29a7ce81c9a654780cf463f8 on Cpaulyz:TsFileTailIndex** into **47fa594403d5bb0c79d8414886b5a9fe6a24b778 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3627: [IOTDB-1517][IOTDB-1521] Refactor TsFile Index for Vector(multi-variable timeseries)
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3627:
URL: https://github.com/apache/iotdb/pull/3627#discussion_r677258393
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/MetadataIndexConstructor.java
##########
@@ -62,49 +68,45 @@ public static MetadataIndexNode constructMetadataIndex(
MetadataIndexNode currentIndexNode =
new MetadataIndexNode(MetadataIndexNodeType.LEAF_MEASUREMENT);
int serializedTimeseriesMetadataNum = 0;
+ String vectorName = ""; // record previous vector name
Review comment:
Remove the field if it is not used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org