[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #910: add system design eng files

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #910: add system design eng files
URL: https://github.com/apache/incubator-iotdb/pull/910#discussion_r392684868
 
 

 ##########
 File path: docs/Documentation/SystemDesign/5-DataQuery/5-GroupByQuery.md
 ##########
 @@ -0,0 +1,260 @@
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+    
+        http://www.apache.org/licenses/LICENSE-2.0
+    
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+-->
+
+# Downsampling query
+
+* org.apache.iotdb.db.query.dataset.groupby.GroupByEngineDataSet
+
+The result set of the downsampling query will inherit `GroupByEngineDataSet`, this class contains the following fields:
+* protected long queryId
+* private long interval
+* private long slidingStep
+
+The following two fields are for the entire query, and the time period is left closed and right open, which is [startTime, endTime):
+* private long startTime
+* private long endTime
+
+The following fields are for the current segment.
+
+* protected long curStartTime;
+* protected long curEndTime;
+* private int usedIndex;
+* protected boolean hasCachedTimeInterval;
+
+
+The core method of `GroupByEngineDataSet` is very easy. First, determine if there is a next segment based on whether there is a cached time period, and return` true`; if not, calculate the segmentation start time and increase `usedIndex` by 1.  If the segment start time has exceeded the query end time, return `false`; otherwise, calculate the query end time, set` hasCachedTimeInterval` to `true`, and return` true`:
+```
+protected boolean hasNextWithoutConstraint() {
+  if (hasCachedTimeInterval) {
+    return true;
+  }
+
+  curStartTime = usedIndex * slidingStep + startTime;
+  usedIndex++;
+  if (curStartTime < endTime) {
+    hasCachedTimeInterval = true;
+    curEndTime = Math.min(curStartTime + interval, endTime);
+    return true;
+  } else {
+    return false;
+  }
+}
+```
+
+## Downsampling query without value filter
+
+The downsampling query logic without value filter is mainly in the `GroupByWithoutValueFilterDataSet` class, which inherits` GroupByEngineDataSet`.
+
+
+This class has the following key fields:
+* private Map <Path, GroupByExecutor> pathExecutors classifies aggregate functions for the same `Path` and encapsulates them as` GroupByExecutor`,
+  `GroupByExecutor` encapsulates the data calculation logic and method of each Path, which will be described later
+
+* private TimeRange timeRange encapsulates the time interval of each calculation into an object, which is used to determine whether Statistics can directly participate in the calculation
+* private Filter timeFilter Generates a user-defined query interval as a Filter object, which is used to filter the available files, chunks, and pages.
+
+
+First, in the initialization `initGroupBy ()` method, the `timeFilter` is calculated based on the expression, and` GroupByExecutor` is generated for each `path`.
+
+The `nextWithoutConstraint ()` method calculates the aggregate value `aggregateResults` of all aggregation methods in each` Path` by calling the `GroupByExecutor.calcResult ()` method.
+The following method is used to convert the result list into a RowRecord. Note that when there are no results in the list, add `null` to the RowRecord:
+
+```
+for (AggregateResult res : fields) {
+  if (res == null) {
+    record.addField(null);
+    continue;
+  }
+  record.addField(res.getResult(), res.getResultDataType());
+}
+```
+
+
+### GroupByExecutor
+Encapsulating the calculation method of all aggregate functions under the same path, this class has the following key fields:
+* private IAggregateReader reader SeriesAggregateReader used to read the current Path data
+* private BatchData preCachedData Every time the data read from `Reader` is a batch, it is likely to exceed the current time period, then this` BatchData` will be cached for next use
+* private List <Pair <AggregateResult, Integer >> results stores all aggregation methods in the current `Path`, for example:` select count (a), sum (a), avg (b) `,` count` and `sum`  The methods are stored together.
+     The `Integer` on the right is used to convert the result set to the order of the user query before converting it to RowRecord.
+
+#### Main method
+
+```
+//Read data from the reader and calculate the main method of this class.
+private List<Pair<AggregateResult, Integer>> calcResult() throws IOException, QueryProcessException;
+
+//Add aggregation operation for current path
+private void addAggregateResult(AggregateResult aggrResult, int index);
+
+//Determine whether the current path has completed all aggregation calculations
+private boolean isEndCalc();
+
+//Calculate results from BatchData that did not run out of cache last calculation
+private boolean calcFromCacheData() throws IOException;
+
+//Calculation using BatchData
+private void calcFromBatch(BatchData batchData) throws IOException;
+
+//Calculate results directly using Page or Chunk's Statistics
+private void calcFromStatistics(Statistics statistics) throws QueryProcessException;
+
+//Clear all calculation results
+private void resetAggregateResults();
+
+//Iterate through and calculate the data in the page
+private boolean readAndCalcFromPage() throws IOException, QueryProcessException;
+
+```
+
+In GroupByExecutor, because different aggregate functions of the same path use the same data, the entry method calcResult is responsible for reading all the data of the Path.
+The retrieved data then calls the calcFromBatch method to complete the calculation of BatchData through all the aggregate functions.
+
+The `calcResult` method returns all AggregateResult under the current Path and the position of the current aggregated value in the user query order. Its main logic is:
+
+```
+//Calculate the data left over from the last time, and end the calculation if you can get the results directly
+if (calcFromCacheData()) {
+    return results;
+}
+
+//Because a chunk contains multiple pages, the page of the current chunk must be used up before the next chunk is opened.
+if (readAndCalcFromPage()) {
+    return results;
+}
+
+//If the remaining data is calculated, open a new chunk to continue the calculation.
+while (reader.hasNextChunk()) {
+    Statistics chunkStatistics = reader.currentChunkStatistics();
+      // Determine if Statistics is available and perform calculations
 
 Review comment:
   there loss of judgment about `chunkStatistics`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services