You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2022/01/21 08:20:06 UTC
[GitHub] [iotdb] RYH61 opened a new pull request #4950: Fix docx ryh
RYH61 opened a new pull request #4950:
URL: https://github.com/apache/iotdb/pull/4950
## Description
### Content1 ...
### Content2 ...
### Content3 ...
<!--
In each section, please describe design decisions made, including:
- Choice of algorithms
- Behavioral aspects. What configuration values are acceptable? How are corner cases and error
conditions handled, such as when there are insufficient resources?
- Class organization and design (how the logic is split between classes, inheritance, composition,
design patterns)
- Method organization and design (how the logic is split between methods, parameters and return types)
- Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
-->
<!-- It's good to describe an alternative design (or mention an alternative name) for every design
(or naming) decision point and compare the alternatives with the designs that you've implemented
(or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
<!-- If there was a discussion of the design of the feature implemented in this PR elsewhere
(e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list),
link to that discussion from this PR description and explain what have changed in your final design
compared to your original proposal or the consensus version in the end of the discussion.
If something hasn't changed since the original discussion, you can omit a detailed discussion of
those aspects of the design here, perhaps apart from brief mentioning for the sake of readability
of this PR description. -->
<!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
<hr>
This PR has:
- [ ] been self-reviewed.
- [ ] concurrent read
- [ ] concurrent write
- [ ] concurrent read and write
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods.
- [ ] added or updated version, __license__, or notice information
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious
for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold
for code coverage.
- [ ] added integration tests.
- [ ] been tested in a test IoTDB cluster.
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items
apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items
from the checklist above are strictly necessary, but it would be very helpful if you at least
self-review the PR. -->
<hr>
##### Key changed/added classes (or packages if there are too many classes) in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] mychaow commented on a change in pull request #4950: Fix docx ryh
Posted by GitBox <gi...@apache.org>.
mychaow commented on a change in pull request #4950:
URL: https://github.com/apache/iotdb/pull/4950#discussion_r790639752
##########
File path: docs/zh/SystemDesign/DataQuery/QueryFundamentals.md
##########
@@ -46,8 +46,7 @@ TsFile 各级结构在前面的 [TsFile](../TsFile/TsFile.md) 文档中已有介
## 顺序和乱序文件的数据特点
-对于顺序和乱序文件的数据，其数据在文件中的分部特征有所不同。
-顺序文件的 TimeseriesMetadata 中所包含的 ChunkMetadata 也是有序的，也就是说如果按照 chunkMetadata1, chunkMetadata2 的顺序存储，那么将会保证 chunkMetadata1.endtime <= chunkMetadata2.startTime。
+对于顺序和乱序文件的数据，其数据在文件中的分部特征有所不同。 顺序文件的 TimeseriesMetadata 中所包含的 ChunkMetadata 也是有序的，也就是说如果按照 chunkMetadata1, chunkMetadata2 的顺序存储，那么将会保证 chunkMetadata1.endtime <= chunkMetadata2.startTime。
Review comment:
Why change two line into one line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] mychaow commented on a change in pull request #4950: Fix docx ryh
Posted by GitBox <gi...@apache.org>.
mychaow commented on a change in pull request #4950:
URL: https://github.com/apache/iotdb/pull/4950#discussion_r790457980
##########
File path: docs/zh/SystemDesign/DataQuery/AggregationQuery.md
##########
@@ -27,68 +27,69 @@
## 不带值过滤条件的聚合查询
-对于不带值过滤条件的聚合查询，通过 `executeWithoutValueFilter()` 方法获得结果并构建 dataSet。首先使用 `mergeSameSeries()` 方法将对于相同时间序列的聚合查询合并，例如：如果需要计算 count(s1), sum(s2), count(s3), sum(s1)，即需要计算 s1 的两个聚合值，那么将会得到 pathToAggrIndexesMap 结果为：s1 -> 0, 3; s2 -> 1; s3 -> 2。
+对于不带值过滤条件的聚合查询，通过 `executeWithoutValueFilter()` 方法获得结果并构建 dataSet。首先使用 `groupAggregationsBySeries` 方法将对于相同时间序列的聚合查询合并，例如：如果需要计算 count(s1), sum(s2), count(s3), sum(s1)，即需要计算 s1 的两个聚合值，那么将会得到 pathToAggrIndexesMap 结果为：s1 -> 0, 3; s2 -> 1; s3 -> 2。
-那么将会得到 `pathToAggrIndexesMap`，其中每一个 entry 都是一个 series 的聚合查询，因此可以通过调用 `groupAggregationsBySeries()` 方法计算出其聚合值 `aggregateResults`。在最后创建结果集之前，需要将其顺序还原为用户查询的顺序。最后使用 `constructDataSet()` 方法创建结果集并返回。
+那么将会得到 `pathToAggrIndexesMap`，其中每一个 entry 都是一个 series 的聚合查询，因此可以通过调用 `aggregateOneSeries` 方法计算出其聚合值 `aggregateResult`。在最后创建结果集之前，需要将其顺序还原为用户查询的顺序。最后使用 `constructDataSet()` 方法创建结果集并返回。
-下面详细讲解 `groupAggregationsBySeries()` 方法。首先创建一个 `IAggregateReader`：
-```
-IAggregateReader seriesReader = new SeriesAggregateReader(
- pathToAggrIndexes.getKey(), tsDataType, context, QueryResourceManager.getInstance()
- .getQueryDataSource(seriesPath, context, timeFilter), timeFilter, null);
+下面详细讲解 `aggregateOneSeries` 方法。首先创建一个 `IAggregateReader`：
+```java
+if (ascAggregateResultList != null && !ascAggregateResultList.isEmpty()) {
+ IAggregateReader seriesReader = new SeriesAggregateReader(
+ seriesPath, measurements, tsDataType, context, queryDataSource, timeFilter,
+ null, null, true);
+ aggregateFromReader(seriesReader, ascAggregateResultList);
+}
+if (descAggregateResultList != null && !descAggregateResultList.isEmpty()) {
+ IAggregateReader seriesReader = new SeriesAggregateReader(
+ seriesPath, measurements, tsDataType, context, queryDataSource, timeFilter,
+ null, null, false);
+ aggregateFromReader(seriesReader, descAggregateResultList);
+}
```
对于每一个 entry（即 series），首先为其每一种聚合查询创建一个聚合结果 `AggregateResult`，同时维护一个布尔值列表 `isCalculatedList`，对应每一个 `AggregateResult`是否已经计算完成，并记录需要剩余计算的聚合函数数目 `remainingToCalculate`。布尔值列表和这个计数值将会使得某些聚合函数（如 `FIRST_VALUE`）在获得结果后，不需要再继续进行整个循环过程。
接下来，按照 5.2 节所介绍的 `aggregateReader` 使用方法，更新 `AggregateResult`：
-```
-while (aggregateReader.hasNextChunk()) {
- if (aggregateReader.canUseCurrentChunkStatistics()) {
- Statistics chunkStatistics = aggregateReader.currentChunkStatistics();
-
- // do some aggregate calculation using chunk statistics
- ...
-
- aggregateReader.skipCurrentChunk();
- continue;
- }
-
- while (aggregateReader.hasNextPage()) {
- if (aggregateReader.canUseCurrentPageStatistics()) {
- Statistics pageStatistic = aggregateReader.currentPageStatistics();
-
- // do some aggregate calculation using page statistics
- ...
-
- aggregateReader.skipCurrentPage();
- continue;
- } else {
- BatchData batchData = aggregateReader.nextPage();
- // do some aggregate calculation using batch data
- ...
- }
- }
-}
-```
+```java
+while (seriesReader.hasNextFile()) {
+ // cal by file statistics
+ if (seriesReader.canUseCurrentFileStatistics()) {
+ Statistics fileStatistics = seriesReader.currentFileStatistics();
+ remainingToCalculate =
+ aggregateStatistics(
+ aggregateResultList, isCalculatedArray, remainingToCalculate, fileStatistics);
+ if (remainingToCalculate == 0) {
+ return;
+ }
+ seriesReader.skipCurrentFile();
+ continue;
+ }
-需要注意的是，在对于每一个 result 进行更新之前，需要首先判断其是否已经被计算完（利用 `isCalculatedList` 列表）；每一次更新后，调用 `isCalculatedAggregationResult()` 方法同时更新列表中的布尔值。如果列表中所有值均为 true，即 `remainingToCalculate` 值为 0，证明所有聚合函数结果均已计算完，可以返回。
-```
-if (Boolean.FALSE.equals(isCalculatedList.get(i))) {
- AggregateResult aggregateResult = aggregateResultList.get(i);
- ... // 更新
- if (aggregateResult.isCalculatedAggregationResult()) {
- isCalculatedList.set(i, true);
- remainingToCalculate--;
- if (remainingToCalculate == 0) {
- return aggregateResultList;
+ while (seriesReader.hasNextChunk()) {
+ // cal by chunk statistics
+ if (seriesReader.canUseCurrentChunkStatistics()) {
+ Statistics chunkStatistics = seriesReader.currentChunkStatistics();
+ remainingToCalculate =
+ aggregateStatistics(
+ aggregateResultList, isCalculatedArray, remainingToCalculate, chunkStatistics);
+ if (remainingToCalculate == 0) {
+ return;
+ }
+ seriesReader.skipCurrentChunk();
+ continue;
+ }
+
+ remainingToCalculate =
+ aggregatePages(
+ seriesReader, aggregateResultList, isCalculatedArray, remainingToCalculate);
+ if (remainingToCalculate == 0) {
+ return;
+ }
}
- }
}
```
-在使用 `overlapedPageData` 进行更新时，由于获得每一个聚合函数结果都会遍历这个 batchData，因此需要调用 `resetBatchData()` 方法将指针指向其开始位置，使得下一个函数可以遍历。
Review comment:
same above
##########
File path: docs/zh/SystemDesign/DataQuery/SeriesReader.md
##########
@@ -163,55 +163,55 @@ Object[] values = readerByTimestamp.getValueInTimestamp(timestamps, length);
```
/*
- * 文件层
- */
-private final List<TsFileResource> seqFileResource;
- 顺序文件列表，因为顺序文件本身就保证有序，且时间戳互不重叠，只需使用 List 进行存储
-
-private final PriorityQueue<TsFileResource> unseqFileResource;
- 乱序文件列表，因为乱序文件互相之间不保证顺序性，且可能有重叠，为了保证顺序，使用优先队列进行存储
+* 文件索引
+*/
Review comment:
this change to be QueryDataResource, not the fileIndex. FileIndex is the index.
##########
File path: docs/zh/SystemDesign/DataQuery/AggregationQuery.md
##########
@@ -27,68 +27,69 @@
## 不带值过滤条件的聚合查询
-对于不带值过滤条件的聚合查询，通过 `executeWithoutValueFilter()` 方法获得结果并构建 dataSet。首先使用 `mergeSameSeries()` 方法将对于相同时间序列的聚合查询合并，例如：如果需要计算 count(s1), sum(s2), count(s3), sum(s1)，即需要计算 s1 的两个聚合值，那么将会得到 pathToAggrIndexesMap 结果为：s1 -> 0, 3; s2 -> 1; s3 -> 2。
+对于不带值过滤条件的聚合查询，通过 `executeWithoutValueFilter()` 方法获得结果并构建 dataSet。首先使用 `groupAggregationsBySeries` 方法将对于相同时间序列的聚合查询合并，例如：如果需要计算 count(s1), sum(s2), count(s3), sum(s1)，即需要计算 s1 的两个聚合值，那么将会得到 pathToAggrIndexesMap 结果为：s1 -> 0, 3; s2 -> 1; s3 -> 2。
-那么将会得到 `pathToAggrIndexesMap`，其中每一个 entry 都是一个 series 的聚合查询，因此可以通过调用 `groupAggregationsBySeries()` 方法计算出其聚合值 `aggregateResults`。在最后创建结果集之前，需要将其顺序还原为用户查询的顺序。最后使用 `constructDataSet()` 方法创建结果集并返回。
+那么将会得到 `pathToAggrIndexesMap`，其中每一个 entry 都是一个 series 的聚合查询，因此可以通过调用 `aggregateOneSeries` 方法计算出其聚合值 `aggregateResult`。在最后创建结果集之前，需要将其顺序还原为用户查询的顺序。最后使用 `constructDataSet()` 方法创建结果集并返回。
-下面详细讲解 `groupAggregationsBySeries()` 方法。首先创建一个 `IAggregateReader`：
-```
-IAggregateReader seriesReader = new SeriesAggregateReader(
- pathToAggrIndexes.getKey(), tsDataType, context, QueryResourceManager.getInstance()
- .getQueryDataSource(seriesPath, context, timeFilter), timeFilter, null);
+下面详细讲解 `aggregateOneSeries` 方法。首先创建一个 `IAggregateReader`：
+```java
+if (ascAggregateResultList != null && !ascAggregateResultList.isEmpty()) {
+ IAggregateReader seriesReader = new SeriesAggregateReader(
+ seriesPath, measurements, tsDataType, context, queryDataSource, timeFilter,
+ null, null, true);
+ aggregateFromReader(seriesReader, ascAggregateResultList);
+}
+if (descAggregateResultList != null && !descAggregateResultList.isEmpty()) {
+ IAggregateReader seriesReader = new SeriesAggregateReader(
+ seriesPath, measurements, tsDataType, context, queryDataSource, timeFilter,
+ null, null, false);
+ aggregateFromReader(seriesReader, descAggregateResultList);
+}
```
对于每一个 entry（即 series），首先为其每一种聚合查询创建一个聚合结果 `AggregateResult`，同时维护一个布尔值列表 `isCalculatedList`，对应每一个 `AggregateResult`是否已经计算完成，并记录需要剩余计算的聚合函数数目 `remainingToCalculate`。布尔值列表和这个计数值将会使得某些聚合函数（如 `FIRST_VALUE`）在获得结果后，不需要再继续进行整个循环过程。
接下来，按照 5.2 节所介绍的 `aggregateReader` 使用方法，更新 `AggregateResult`：
-```
-while (aggregateReader.hasNextChunk()) {
- if (aggregateReader.canUseCurrentChunkStatistics()) {
- Statistics chunkStatistics = aggregateReader.currentChunkStatistics();
-
- // do some aggregate calculation using chunk statistics
- ...
-
- aggregateReader.skipCurrentChunk();
- continue;
- }
-
- while (aggregateReader.hasNextPage()) {
- if (aggregateReader.canUseCurrentPageStatistics()) {
- Statistics pageStatistic = aggregateReader.currentPageStatistics();
-
- // do some aggregate calculation using page statistics
- ...
-
- aggregateReader.skipCurrentPage();
- continue;
- } else {
- BatchData batchData = aggregateReader.nextPage();
- // do some aggregate calculation using batch data
- ...
- }
- }
-}
-```
+```java
+while (seriesReader.hasNextFile()) {
+ // cal by file statistics
+ if (seriesReader.canUseCurrentFileStatistics()) {
+ Statistics fileStatistics = seriesReader.currentFileStatistics();
+ remainingToCalculate =
+ aggregateStatistics(
+ aggregateResultList, isCalculatedArray, remainingToCalculate, fileStatistics);
+ if (remainingToCalculate == 0) {
+ return;
+ }
+ seriesReader.skipCurrentFile();
+ continue;
+ }
-需要注意的是，在对于每一个 result 进行更新之前，需要首先判断其是否已经被计算完（利用 `isCalculatedList` 列表）；每一次更新后，调用 `isCalculatedAggregationResult()` 方法同时更新列表中的布尔值。如果列表中所有值均为 true，即 `remainingToCalculate` 值为 0，证明所有聚合函数结果均已计算完，可以返回。
Review comment:
Why delete the description here
##########
File path: docs/zh/SystemDesign/DataQuery/QueryFundamentals.md
##########
@@ -47,9 +47,9 @@ TsFile 各级结构在前面的 [TsFile](../TsFile/TsFile.md) 文档中已有介
## 顺序和乱序文件的数据特点
对于顺序和乱序文件的数据，其数据在文件中的分部特征有所不同。
-顺序文件的 TimeseriesMetadata 中所包含的 ChunkMetadata 也是有序的，也就是说如果按照 chunkMetadata1, chunkMetadata2 的顺序存储，那么将会保证 chunkMetadata1.endtime <= chunkMetadata2.startTime。
+顺序文件的 TimeseriesIndex 中所包含的 ChunkIndex 也是有序的，也就是说如果按照 ChunkIndex1, ChunkIndex2 的顺序存储，那么将会保证 ChunkIndex1.endtime <= ChunkIndex2.startTime。
Review comment:
I do not find the timeseriesindex class, could you confirm it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] mychaow merged pull request #4950: Update documents in SystemDesign
Posted by GitBox <gi...@apache.org>.
mychaow merged pull request #4950:
URL: https://github.com/apache/iotdb/pull/4950
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org