You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iotdb.apache.org by qi...@apache.org on 2020/06/05 03:51:40 UTC
[incubator-iotdb] branch master updated: Update Last query design
documentation (#1312)
This is an automated email from the ASF dual-hosted git repository.
qiaojialin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-iotdb.git
The following commit(s) were added to refs/heads/master by this push:
new b54923e Update Last query design documentation (#1312)
b54923e is described below
commit b54923e34d882f7c4516496d5e23ae05a87c4a4a
Author: wshao08 <59...@users.noreply.github.com>
AuthorDate: Fri Jun 5 11:51:33 2020 +0800
Update Last query design documentation (#1312)
* Fix bug when unseq last timestamp larger than seq last timestamp
* update docs
---
docs/SystemDesign/DataQuery/LastQuery.md | 67 +++++++++++++---------
docs/zh/SystemDesign/DataQuery/LastQuery.md | 66 ++++++++++++---------
.../iotdb/db/query/executor/LastQueryExecutor.java | 5 +-
.../apache/iotdb/db/integration/IoTDBLastIT.java | 44 ++++++++++++++
4 files changed, 128 insertions(+), 54 deletions(-)
diff --git a/docs/SystemDesign/DataQuery/LastQuery.md b/docs/SystemDesign/DataQuery/LastQuery.md
index 7a8aeba..39cf379 100644
--- a/docs/SystemDesign/DataQuery/LastQuery.md
+++ b/docs/SystemDesign/DataQuery/LastQuery.md
@@ -47,40 +47,55 @@ If it is found that the cache has not been written, execute the following standa
## Last standard query process
-Last standard query process needs to traverse all sequential files and unsequential files to get query results, and finally write the query results back to the MNode cache. In the algorithm, sequential files and unsequential files are processed separately.
-- The sequential file is sorted by its writing time, so use the `loadChunkMetadataFromTsFileResource` method directly to get the last` ChunkMetadata`, and get the maximum timestamp and corresponding value through the statistical data of `ChunkMetadata`.
+Last standard query process needs to scan sequential files and unsequential files in a reverse order to get query result, and finally write the query result back to the MNode cache. In the algorithm, sequential files and unsequential files are processed separately.
+- The sequential files are sorted by its writing time, so use the `loadTimeSeriesMetadata()` method directly to get the last valid ` TimeseriesMetadata`. If the statistics of `TimeseriesMetadata` is available, the Last pair could be returned directly, otherwise we need to call `loadChunkMetadataList()` to get the last `ChunkMetadata` structure and obtain the Last time-value pair via the statistical data of `ChunkMetadata`.
```
- if (!seqFileResources.isEmpty()) {
- List<ChunkMetaData> chunkMetadata =
- FileLoaderUtils.loadChunkMetadataFromTsFileResource(
- seqFileResources.get(seqFileResources.size() - 1), seriesPath, context);
- if (!chunkMetadata.isEmpty()) {
- ChunkMetaData lastChunkMetaData = chunkMetadata.get(chunkMetadata.size() - 1);
- Statistics chunkStatistics = lastChunkMetaData.getStatistics();
- resultPair =
- constructLastPair(
- chunkStatistics.getEndTime(), chunkStatistics.getLastValue(), tsDataType);
+ for (int i = seqFileResources.size() - 1; i >= 0; i--) {
+ TimeseriesMetadata timeseriesMetadata = FileLoaderUtils.loadTimeSeriesMetadata(
+ seqFileResources.get(i), seriesPath, context, null, sensors);
+ if (timeseriesMetadata != null) {
+ if (!timeseriesMetadata.isModified()) {
+ Statistics timeseriesMetadataStats = timeseriesMetadata.getStatistics();
+ resultPair = constructLastPair(
+ timeseriesMetadataStats.getEndTime(),
+ timeseriesMetadataStats.getLastValue(),
+ tsDataType);
+ break;
+ } else {
+ List<ChunkMetadata> chunkMetadataList = timeseriesMetadata.loadChunkMetadataList();
+ if (!chunkMetadataList.isEmpty()) {
+ ChunkMetadata lastChunkMetaData = chunkMetadataList.get(chunkMetadataList.size() - 1);
+ Statistics chunkStatistics = lastChunkMetaData.getStatistics();
+ resultPair =
+ constructLastPair(
+ chunkStatistics.getEndTime(), chunkStatistics.getLastValue(), tsDataType);
+ break;
+ }
+ }
+ }
}
- }
```
-- Unsequential files need to traverse all `ChunkMetadata` structures to get the maximum timestamp data. It should be noted that when multiple `ChunkMetadata` have the same timestamp, we take the data in` ChunkMatadata` with the largest `version` value as the result of Last.
+- For unsequential files, we need to traverse all valid `TimeseriesMetadata` structures and keep updating the current Last timestamp to find the biggest timestamp. It should be noted that when multiple `ChunkMetadata` have the same timestamp, we take the data in` ChunkMatadata` with the largest `version` value as the result of Last.
```
long version = 0;
for (TsFileResource resource : unseqFileResources) {
- if (resource.getEndTimeMap().get(seriesPath.getDevice()) < resultPair.getTimestamp()) {
- break;
+ if (resource.getEndTime(seriesPath.getDevice()) < resultPair.getTimestamp()) {
+ continue;
}
- List<ChunkMetaData> chunkMetadata =
- FileLoaderUtils.loadChunkMetadataFromTsFileResource(resource, seriesPath, context);
- for (ChunkMetaData chunkMetaData : chunkMetadata) {
- if (chunkMetaData.getEndTime() == resultPair.getTimestamp()
- && chunkMetaData.getVersion() > version) {
- Statistics chunkStatistics = chunkMetaData.getStatistics();
- resultPair =
- constructLastPair(
- chunkStatistics.getEndTime(), chunkStatistics.getLastValue(), tsDataType);
- version = chunkMetaData.getVersion();
+ TimeseriesMetadata timeseriesMetadata =
+ FileLoaderUtils.loadTimeSeriesMetadata(resource, seriesPath, context, null, sensors);
+ if (timeseriesMetadata != null) {
+ for (ChunkMetadata chunkMetaData : timeseriesMetadata.loadChunkMetadataList()) {
+ if (chunkMetaData.getEndTime() > resultPair.getTimestamp()
+ || (chunkMetaData.getEndTime() == resultPair.getTimestamp()
+ && chunkMetaData.getVersion() > version)) {
+ Statistics chunkStatistics = chunkMetaData.getStatistics();
+ resultPair =
+ constructLastPair(
+ chunkStatistics.getEndTime(), chunkStatistics.getLastValue(), tsDataType);
+ version = chunkMetaData.getVersion();
+ }
}
}
}
diff --git a/docs/zh/SystemDesign/DataQuery/LastQuery.md b/docs/zh/SystemDesign/DataQuery/LastQuery.md
index 3394442..dfc252e 100644
--- a/docs/zh/SystemDesign/DataQuery/LastQuery.md
+++ b/docs/zh/SystemDesign/DataQuery/LastQuery.md
@@ -44,40 +44,54 @@ if (((LeafMNode) node).getCachedLast() != null) {
## Last标准查询流程
-Last标准查询流程需要遍历所有的顺序文件和乱序文件得到查询结果，最后将查询结果写回到MNode缓存。算法中对顺序文件和乱序文件分别进行处理。
-- 顺序文件由于是对其写入时间已经排好序，因此直接使用`loadChunkMetadataFromTsFileResource`方法取出最后一个`ChunkMetadata`，通过`ChunkMetadata`的统计数据得到最大时间戳和对应的值。
+Last标准查询流程需要以倒序方式扫描顺序文件和乱序文件，一旦得到满足要求的Last查询结果就将其写回到MNode缓存中并返回。算法中对顺序文件和乱序文件分别进行处理。
+- 顺序文件由于是按照写入时间已经排好序，因此直接使用`loadTimeSeriesMetadata()`方法取出最后一个不为空的`TimeseriesMetadata`。若`TimeseriesMetadata`的统计数据可用即可直接得到Last时间戳和对应的值；如不可用则需要使用`loadChunkMetadataList()`方法得到下一层的最后一个`ChunkMetadata`，通过统计数据得到Last结果。
```
- if (!seqFileResources.isEmpty()) {
- List<ChunkMetaData> chunkMetadata =
- FileLoaderUtils.loadChunkMetadataFromTsFileResource(
- seqFileResources.get(seqFileResources.size() - 1), seriesPath, context);
- if (!chunkMetadata.isEmpty()) {
- ChunkMetaData lastChunkMetaData = chunkMetadata.get(chunkMetadata.size() - 1);
- Statistics chunkStatistics = lastChunkMetaData.getStatistics();
- resultPair =
- constructLastPair(
- chunkStatistics.getEndTime(), chunkStatistics.getLastValue(), tsDataType);
+ for (int i = seqFileResources.size() - 1; i >= 0; i--) {
+ TimeseriesMetadata timeseriesMetadata = FileLoaderUtils.loadTimeSeriesMetadata(
+ seqFileResources.get(i), seriesPath, context, null, sensors);
+ if (timeseriesMetadata != null) {
+ if (!timeseriesMetadata.isModified()) {
+ Statistics timeseriesMetadataStats = timeseriesMetadata.getStatistics();
+ resultPair = constructLastPair(
+ timeseriesMetadataStats.getEndTime(),
+ timeseriesMetadataStats.getLastValue(),
+ tsDataType);
+ break;
+ } else {
+ List<ChunkMetadata> chunkMetadataList = timeseriesMetadata.loadChunkMetadataList();
+ if (!chunkMetadataList.isEmpty()) {
+ ChunkMetadata lastChunkMetaData = chunkMetadataList.get(chunkMetadataList.size() - 1);
+ Statistics chunkStatistics = lastChunkMetaData.getStatistics();
+ resultPair =
+ constructLastPair(
+ chunkStatistics.getEndTime(), chunkStatistics.getLastValue(), tsDataType);
+ break;
+ }
+ }
+ }
}
- }
```
-- 乱序文件则需要遍历所有的`ChunkMetadata`结构得到最大时间戳数据。需要注意的是当多个`ChunkMetadata`拥有相同的时间戳时，我们取`version`值最大的`ChunkMatadata`中的数据作为Last的结果。
+- 对于乱序文件，需要遍历所有不为空的`TimeseriesMetadata`结构并更新当前最大时间戳的Last数据，直到扫描完所有乱序文件为止。需要注意的是当多个`ChunkMetadata`拥有相同的最大时间戳时，我们取`version`值最大的`ChunkMatadata`中的数据作为Last的结果。
```
long version = 0;
for (TsFileResource resource : unseqFileResources) {
- if (resource.getEndTimeMap().get(seriesPath.getDevice()) < resultPair.getTimestamp()) {
- break;
+ if (resource.getEndTime(seriesPath.getDevice()) < resultPair.getTimestamp()) {
+ continue;
}
- List<ChunkMetaData> chunkMetadata =
- FileLoaderUtils.loadChunkMetadataFromTsFileResource(resource, seriesPath, context);
- for (ChunkMetaData chunkMetaData : chunkMetadata) {
- if (chunkMetaData.getEndTime() == resultPair.getTimestamp()
- && chunkMetaData.getVersion() > version) {
- Statistics chunkStatistics = chunkMetaData.getStatistics();
- resultPair =
- constructLastPair(
- chunkStatistics.getEndTime(), chunkStatistics.getLastValue(), tsDataType);
- version = chunkMetaData.getVersion();
+ TimeseriesMetadata timeseriesMetadata =
+ FileLoaderUtils.loadTimeSeriesMetadata(resource, seriesPath, context, null, sensors);
+ if (timeseriesMetadata != null) {
+ for (ChunkMetadata chunkMetaData : timeseriesMetadata.loadChunkMetadataList()) {
+ if (chunkMetaData.getEndTime() == resultPair.getTimestamp()
+ && chunkMetaData.getVersion() > version) {
+ Statistics chunkStatistics = chunkMetaData.getStatistics();
+ resultPair =
+ constructLastPair(
+ chunkStatistics.getEndTime(), chunkStatistics.getLastValue(), tsDataType);
+ version = chunkMetaData.getVersion();
+ }
}
}
}
diff --git a/server/src/main/java/org/apache/iotdb/db/query/executor/LastQueryExecutor.java b/server/src/main/java/org/apache/iotdb/db/query/executor/LastQueryExecutor.java
index 33ab8a4..7b7fb08 100644
--- a/server/src/main/java/org/apache/iotdb/db/query/executor/LastQueryExecutor.java
+++ b/server/src/main/java/org/apache/iotdb/db/query/executor/LastQueryExecutor.java
@@ -166,8 +166,9 @@ public class LastQueryExecutor {
FileLoaderUtils.loadTimeSeriesMetadata(resource, seriesPath, context, null, sensors);
if (timeseriesMetadata != null) {
for (ChunkMetadata chunkMetaData : timeseriesMetadata.loadChunkMetadataList()) {
- if (chunkMetaData.getEndTime() == resultPair.getTimestamp()
- && chunkMetaData.getVersion() > version) {
+ if (chunkMetaData.getEndTime() > resultPair.getTimestamp()
+ || (chunkMetaData.getEndTime() == resultPair.getTimestamp()
+ && chunkMetaData.getVersion() > version)) {
Statistics chunkStatistics = chunkMetaData.getStatistics();
resultPair =
constructLastPair(
diff --git a/server/src/test/java/org/apache/iotdb/db/integration/IoTDBLastIT.java b/server/src/test/java/org/apache/iotdb/db/integration/IoTDBLastIT.java
index 7093afd..ee2c843 100644
--- a/server/src/test/java/org/apache/iotdb/db/integration/IoTDBLastIT.java
+++ b/server/src/test/java/org/apache/iotdb/db/integration/IoTDBLastIT.java
@@ -268,6 +268,50 @@ public class IoTDBLastIT {
}
}
+ @Test
+ public void lastWithUnseqTimeLargerThanSeqTimeTest() throws SQLException, MetadataException {
+ String[] retArray =
+ new String[] {
+ "150,root.ln.wf01.wt04.temperature,31.2",
+ };
+
+
+ try (Connection connection =
+ DriverManager.getConnection("jdbc:iotdb://127.0.0.1:6667/", "root", "root");
+ Statement statement = connection.createStatement()) {
+
+ statement.execute("SET STORAGE GROUP TO root.ln.wf01.wt04");
+ statement.execute("CREATE TIMESERIES root.ln.wf01.wt04.status WITH DATATYPE=BOOLEAN, ENCODING=PLAIN");
+ statement.execute("CREATE TIMESERIES root.ln.wf01.wt04.temperature WITH DATATYPE=DOUBLE, ENCODING=PLAIN");
+ statement.execute("INSERT INTO root.ln.wf01.wt04(timestamp,temperature) values(100,22.1)");
+ statement.execute("flush");
+ statement.execute("INSERT INTO root.ln.wf01.wt04(timestamp,status) values(200,true)");
+ statement.execute("flush");
+ statement.execute("INSERT INTO root.ln.wf01.wt04(timestamp,temperature) values(150,31.2)");
+ statement.execute("flush");
+
+ MNode node = MManager.getInstance().getNodeByPath("root.ln.wf01.wt03.temperature");
+ ((LeafMNode) node).resetCache();
+
+ boolean hasResultSet = statement.execute(
+ "select last temperature from root.ln.wf01.wt04");
+
+ Assert.assertTrue(hasResultSet);
+ int cnt = 0;
+ try (ResultSet resultSet = statement.getResultSet()) {
+ while (resultSet.next()) {
+ String ans =
+ resultSet.getString(TIMESTAMP_STR) + ","
+ + resultSet.getString(TIMESEIRES_STR) + ","
+ + resultSet.getString(VALUE_STR);
+ Assert.assertEquals(retArray[cnt], ans);
+ cnt++;
+ }
+ Assert.assertEquals(cnt, 1);
+ }
+ }
+ }
+
private void prepareData() {
try (Connection connection = DriverManager
.getConnection(Config.IOTDB_URL_PREFIX + "127.0.0.1:6667/", "root",