[GitHub] [incubator-iotdb] liutaohua opened a new pull request #692: add some impl

[GitBox <gi...@apache.org>]

liutaohua opened a new pull request #692: add some impl
URL: https://github.com/apache/incubator-iotdb/pull/692
 
 
   still woking 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on issue #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on issue #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#issuecomment-573270942
 
 
   > please try to merge master to check whether the failed test can pass.
   
   please help me merge master to IoTDB  new_series_reader 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365772754
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > unSeqStartTime) {
 
 Review comment:
   ```suggestion
         if (chunkMetaData.getEndTime() >= unSeqStartTime) {
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365672791
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/chunkRelated/ChunkReaderIterator.java
 ##########
 @@ -34,12 +35,12 @@
  * are used here: <code>ChunkReaderWithoutFilter</code> and <code>ChunkReaderWithFilter</code>.
  * <p>
  */
-public class DiskChunkReader implements IPointReader, IBatchReader {
+public class ChunkReaderIterator implements IPointReader, IBatchReader {
 
 Review comment:
   rename to ChunkDataIterator

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365566073
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/SeriesDataReaderWithoutValueFilter.java
 ##########
 @@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+import java.io.IOException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.statistics.Statistics;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+
+/**
+ * @Author: LiuDaWei
+ * @Create: 2020年01月05日
+ */
+public class SeriesDataReaderWithoutValueFilter extends AbstractDataReader implements
+    IAggregateReader {
+
+  public SeriesDataReaderWithoutValueFilter(Path seriesPath, TSDataType dataType, Filter timeFilter,
+      QueryContext context) throws StorageEngineException, IOException {
+    super(seriesPath, dataType, timeFilter, context);
+  }
+
+
+  @Override
+  public boolean hasNextChunk() throws IOException {
+    return super.hasNextChunk();
+  }
+
+  public boolean canUseChunkStatistics() {
+    Statistics statistics = chunkMetaData.getStatistics();
+    return !overlappedChunkReader.isEmpty() && canUseStatistics(statistics);
+  }
+
+  @Override
+  public Statistics currentChunkStatistics() {
+    return chunkMetaData.getStatistics();
+  }
+
+  @Override
+  public void skipChunkData() throws IOException {
+    hasCachedNextChunk = false;
+  }
+
+
+  @Override
+  public boolean hasNextPage() throws IOException {
+    return super.hasNextPage();
+  }
+
+  @Override
+  public boolean canUsePageStatistics() {
+    Statistics pageStatistics = currentPage.getStatistics();
+    return !overlappedPages.isEmpty() && canUseStatistics(pageStatistics);
+  }
+
+  @Override
+  public Statistics currentPageStatistics() {
+    return currentPage.getStatistics();
+  }
+
+  @Override
+  public void skipPageData() throws IOException {
+    hasCachedNextPage = false;
+  }
+
+
+  @Override
+  public boolean hasNextBatch() throws IOException {
+    return super.hasNextBatch();
+  }
+
+  @Override
+  public BatchData nextBatch() throws IOException {
+    return super.nextBatch();
+  }
+
+
+  protected boolean canUseStatistics(Statistics statistics) {
+    if (filter == null || !filter.containStartEndTime(statistics.getStartTime(),
+        statistics.getEndTime())) {
+      return false;
+    }
+    return true;
 
 Review comment:
   can be simplified to `return filter != null && filter.containStartEndTime(statistics.getStartTime(), statistics.getEndTime());`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365768651
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > unSeqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+    while (!seqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = seqFileResource.get(0).getStartTimeMap();
+      Long seqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > seqStartTime) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * Because seq data and unseq data intersect, the minimum startTime taken from two files at a time
+   * is used as the reference time to start reading data
+   */
+  private void searchMinimumChunk() {
+    hasCachedNextChunk = true;
+    if (!seqChunkMetadatas.isEmpty() && unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = seqChunkMetadatas.remove(0);
+    } else if (seqChunkMetadatas.isEmpty() && !unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = unseqChunkMetadatas.pollFirst();
+    } else if (!seqChunkMetadatas.isEmpty()) {
+      // seq 和 unseq 的 chunk metadata 都不为空
+      if (seqChunkMetadatas.get(0).getStartTime() <= unseqChunkMetadatas.first().getStartTime()) {
+        chunkMetaData = seqChunkMetadatas.remove(0);
+      } else {
+        chunkMetaData = unseqChunkMetadatas.pollFirst();
+      }
+    } else {
+      hasCachedNextChunk = false;
+    }
+  }
+
+  /**
+   * Before reading the chunks, should first clean up all the useless chunks, because in the file
+   * hierarchy, although the files are available, some of the internal chunks are still unavailable
+   *
+   * @throws IOException
+   */
+  private void removeInvalidChunks() throws IOException {
+    //remove seq chunks
+    while (filter != null && (!seqChunkMetadatas.isEmpty() || !seqFileResource.isEmpty())) {
+      while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+      }
+
+      ChunkMetaData metaData = seqChunkMetadatas.get(0);
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        seqChunkMetadatas.remove(0);
+        continue;
+      }
+      break;
+    }
+    while (filter != null && (!unseqChunkMetadatas.isEmpty() || !unseqFileResource.isEmpty())) {
+      while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+      }
+      ChunkMetaData metaData = unseqChunkMetadatas.first();
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        unseqChunkMetadatas.pollFirst();
+        continue;
+      }
+      break;
+    }
+    fillMetadataContainer();
 
 Review comment:
   is this necessary?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365637657
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/executor/NewAggregateEngineExecutor.java
 ##########
 @@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.query.executor;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.path.PathException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.MManager;
+import org.apache.iotdb.db.qp.physical.crud.AggregationPlan;
+import org.apache.iotdb.db.query.aggregation.AggreResultData;
+import org.apache.iotdb.db.query.aggregation.AggregateFunction;
+import org.apache.iotdb.db.query.aggregation.impl.LastValueAggrFunc;
+import org.apache.iotdb.db.query.aggregation.impl.MaxTimeAggrFunc;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.dataset.AggreResultDataPointReader;
+import org.apache.iotdb.db.query.dataset.OldEngineDataSetWithoutValueFilter;
+import org.apache.iotdb.db.query.factory.AggreFuncFactory;
+import org.apache.iotdb.db.query.reader.IPointReader;
+import org.apache.iotdb.db.query.reader.IReaderByTimestamp;
+import org.apache.iotdb.db.query.reader.seriesRelated.SeriesDataReaderWithoutValueFilter;
+import org.apache.iotdb.db.query.reader.seriesRelated.SeriesReaderByTimestamp;
+import org.apache.iotdb.db.query.timegenerator.EngineTimeGenerator;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.statistics.Statistics;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.expression.IExpression;
+import org.apache.iotdb.tsfile.read.expression.impl.GlobalTimeExpression;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+public class NewAggregateEngineExecutor {
+
+  private List<Path> selectedSeries;
+  private List<String> aggres;
+  private IExpression expression;
+
+  /**
+   * aggregation batch calculation size.
+   **/
+  private int aggregateFetchSize;
+
+  /**
+   * constructor.
+   */
+  public NewAggregateEngineExecutor(AggregationPlan aggregationPlan) {
+    this.selectedSeries = aggregationPlan.getDeduplicatedPaths();
+    this.aggres = aggregationPlan.getDeduplicatedAggregations();
+    this.expression = aggregationPlan.getExpression();
+    this.aggregateFetchSize = IoTDBDescriptor.getInstance().getConfig().getBatchSize();
+  }
+
+  /**
+   * execute aggregate function with only time filter or no filter.
+   *
+   * @param context query context
+   */
+  public QueryDataSet executeWithoutValueFilter(QueryContext context)
+      throws StorageEngineException, IOException, QueryProcessException {
+    Filter timeFilter = null;
+    if (expression != null) {
+      timeFilter = ((GlobalTimeExpression) expression).getFilter();
+    }
+
+    List<SeriesDataReaderWithoutValueFilter> readersOfSequenceData = new ArrayList<>();
+    List<AggregateFunction> aggregateFunctions = new ArrayList<>();
+    for (int i = 0; i < selectedSeries.size(); i++) {
+      // construct AggregateFunction
+      TSDataType tsDataType = MManager.getInstance()
+          .getSeriesType(selectedSeries.get(i).getFullPath());
+      AggregateFunction function = AggreFuncFactory.getAggrFuncByName(aggres.get(i), tsDataType);
+      function.init();
+      aggregateFunctions.add(function);
+
+      // sequence reader for sealed tsfile, unsealed tsfile, memory
+      SeriesDataReaderWithoutValueFilter newSeriesReaderWithoutValueFilter = new SeriesDataReaderWithoutValueFilter(
+          selectedSeries.get(i), tsDataType, timeFilter, context);
+      readersOfSequenceData.add(newSeriesReaderWithoutValueFilter);
+    }
+
+    List<AggreResultData> aggreResultDataList = new ArrayList<>();
+    //TODO use multi-thread
+    for (int i = 0; i < selectedSeries.size(); i++) {
+      AggreResultData aggreResultData = aggregateWithoutValueFilter(aggregateFunctions.get(i),
+          readersOfSequenceData.get(i), timeFilter);
+      aggreResultDataList.add(aggreResultData);
+    }
+    return constructDataSet(aggreResultDataList);
+  }
+
+  /**
+   * calculation aggregate result with only time filter or no filter for one series.
+   *
+   * @return one series aggregate result data
+   */
+  private AggreResultData aggregateWithoutValueFilter(AggregateFunction function,
+      SeriesDataReaderWithoutValueFilter newSeriesReader, Filter filter)
+      throws IOException, QueryProcessException {
+    if (function instanceof MaxTimeAggrFunc || function instanceof LastValueAggrFunc) {
+      return handleLastMaxTimeWithOutTimeGenerator(function, newSeriesReader, filter);
+    }
+
+    while (newSeriesReader.hasNextChunk()) {
+      if (newSeriesReader.canUseChunkStatistics()) {
+        Statistics chunkStatistics = newSeriesReader.currentChunkStatistics();
+        function.calculateValueFromStatistics(chunkStatistics);
+        if (function.isCalculatedAggregationResult()) {
+          return function.getResult();
+        }
+        newSeriesReader.skipChunkData();
+        continue;
+      }
+      while (newSeriesReader.hasNextPage()) {
+        //cal by pageheader
+        if (newSeriesReader.canUsePageStatistics()) {
+          Statistics pageStatistic = newSeriesReader.currentChunkStatistics();
+          function.calculateValueFromStatistics(pageStatistic);
+          if (function.isCalculatedAggregationResult()) {
+            return function.getResult();
+          }
+          newSeriesReader.skipPageData();
+          continue;
+        }
+        //cal by pagedata
+        while (newSeriesReader.hasNextBatch()) {
+          function.calculateValueFromPageData(newSeriesReader.nextBatch());
+          if (function.isCalculatedAggregationResult()) {
+            return function.getResult();
+          }
+        }
+      }
+    }
+    return function.getResult();
 
 Review comment:
   extract these codes to avoid repeated codes in this file

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365575434
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.DiskChunkReader;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      this.filter = filter;
+    }
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  protected boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new DiskChunkReader(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader.addReaderWithPriority(new DiskChunkReader(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  protected BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    return null;
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   *
+   * @param seriesPath
+   * @param filter
+   */
+  private void removeInvalidFiles(Path seriesPath, Filter filter) {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      TsFileResource tsFileResource = seqFileResource.get(0);
+      long startTime = tsFileResource.getStartTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      long endTime = tsFileResource.getEndTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+      if (!filter.satisfyStartEndTime(startTime, endTime)) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      TsFileResource tsFileResource = unseqFileResource.first();
+      Long startTime = tsFileResource.getStartTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      Long endTime = tsFileResource.getEndTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+      if (!filter.satisfyStartEndTime(startTime, endTime)) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > unSeqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+    while (!seqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = seqFileResource.get(0).getStartTimeMap();
+      Long seqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > seqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * Because seq data and unseq data intersect, the minimum startTime taken from two files at a time
+   * is used as the reference time to start reading data
+   */
+  private void searchMinimumChunk() {
+    hasCachedNextChunk = true;
+    if (!seqChunkMetadatas.isEmpty() && unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = seqChunkMetadatas.remove(0);
+    } else if (seqChunkMetadatas.isEmpty() && !unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = unseqChunkMetadatas.pollFirst();
+    } else if (!seqChunkMetadatas.isEmpty()) {
+      // seq 和 unseq 的 chunk metadata 都不为空
+      if (seqChunkMetadatas.get(0).getStartTime() <= unseqChunkMetadatas.first().getStartTime()) {
+        chunkMetaData = seqChunkMetadatas.remove(0);
+      } else {
+        chunkMetaData = unseqChunkMetadatas.pollFirst();
+      }
+    } else {
+      hasCachedNextChunk = false;
+    }
+  }
+
+  /**
+   * Before reading the chunks, should first clean up all the useless chunks, because in the file
+   * hierarchy, although the files are available, some of the internal chunks are still unavailable
+   *
+   * @throws IOException
+   */
+  private void removeInvalidChunks() throws IOException {
+    //remove seq chunks
+    while (filter != null && (!seqChunkMetadatas.isEmpty() || !seqFileResource.isEmpty())) {
+      while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+      }
+
+      ChunkMetaData metaData = seqChunkMetadatas.get(0);
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        seqChunkMetadatas.remove(0);
+        continue;
+      }
+      break;
+    }
+    while (filter != null && (!unseqChunkMetadatas.isEmpty() || !unseqFileResource.isEmpty())) {
+      while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+      }
+      ChunkMetaData metaData = unseqChunkMetadatas.first();
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        unseqChunkMetadatas.pollFirst();
+        continue;
+      }
+      break;
+    }
+    fillMetadataContainer();
+  }
+
+  /**
+   * Because there may be too many files in the scenario used by the user, we cannot open all the
+   * chunks at once, which may OOM, so we can only fill one file at a time when needed. This
+   * approach is likely to be ubiquitous, but it keeps the system running smoothly
+   *
+   * @throws IOException
+   */
+  private void fillMetadataContainer() throws IOException {
+    while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+      seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+    }
+    while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+      unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+    }
+  }
+
+  /**
+   * Before calling this method, you should make sure that all the intersecting files are filled in
+   * the container, because the files intersect, but some chunks may still be useless, so you need
+   * to clean up all the unused chunks and populate the container. It should be noted that this
+   * chunk collection is not in order, and all chunks should be used at once
+   */
+  private void fillOverlappedChunks() throws IOException {
+    while (!unseqChunkMetadatas.isEmpty()) {
+      long startTime = unseqChunkMetadatas.first().getStartTime();
+
+      if (chunkMetaData.getEndTime() > startTime) {
+        ChunkMetaData metaData = unseqChunkMetadatas.pollFirst();
+        IChunkReader chunkReader = initChunkReader(metaData);
+        //When data points overlap, there should be a weight
+        overlappedChunkReader.add(new VersionPair<>(metaData.getVersion(), chunkReader));
+        continue;
+      }
+      break;
+    }
+    while (!seqChunkMetadatas.isEmpty()) {
+      long startTime = seqChunkMetadatas.get(0).getStartTime();
+
+      if (chunkMetaData.getEndTime() > startTime) {
+        ChunkMetaData metaData = seqChunkMetadatas.remove(0);
+        IChunkReader chunkReader = initChunkReader(metaData);
+        overlappedChunkReader.add(new VersionPair<>(metaData.getVersion(), chunkReader));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * This is just a fake page container, because no matter how chaotic the situation is, the data in
+   * a chunk is always in order, so when the first page of the chunk cannot be used, there is no
+   * need to look at the whole chunk.
+   */
+  private void fillOverlappedPages() {
+    currentPage = chunkReader.nextPageHeader();
+    hasCachedNextPage = true;
+    latestDirectlyOverlappedPageEndTime = currentPage.getEndTime();
+    while (!overlappedChunkReader.isEmpty()) {
+      VersionPair<IChunkReader> iChunkReader = overlappedChunkReader.get(0);
+      if (currentPage.getEndTime() > iChunkReader.data.nextPageHeader().getStartTime()) {
+        overlappedPages.add(overlappedChunkReader.remove(0));
+      }
+    }
+  }
+
+  private class VersionPair<T> {
+
+    private long version;
+    private T data;
+
+    public VersionPair(long version, T data) {
+      this.version = version;
+      this.data = data;
+    }
+  }
+
+  public void close() throws IOException {
+    if (chunkMetaData != null) {
+      chunkMetaData.getChunkLoader().close();
+    }
+    for (int i = 0; i < openedChunkLoaders.size(); i++) {
+      openedChunkLoaders.get(i).close();
+    }
 
 Review comment:
   could use for-each to avoid variable `i`:
   ```
   for (IChunkLoader openedChunkLoader : openedChunkLoaders) {
         openedChunkLoader.close();
   }
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365649304
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java
 ##########
 @@ -606,7 +606,8 @@ public String getStorageGroupName() {
       }
       // memSeriesLazyMerger has handled the props,
       // so we do not need to handle it again in the following readOnlyMemChunk
-      ReadOnlyMemChunk timeValuePairSorter = new ReadOnlyMemChunk(dataType, memSeriesLazyMerger,
+      ReadOnlyMemChunk timeValuePairSorter = new ReadOnlyMemChunk(measurementId, dataType,
+          memSeriesLazyMerger,
           Collections.emptyMap());
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365788668
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365564437
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/SeriesDataReaderWithoutValueFilter.java
 ##########
 @@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+import java.io.IOException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.statistics.Statistics;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+
+/**
+ * @Author: LiuDaWei
+ * @Create: 2020年01月05日
+ */
 
 Review comment:
   Don't forget to remove it : )

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365648858
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/IAggregateReader.java
 ##########
 @@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+import java.io.IOException;
+import org.apache.iotdb.tsfile.file.metadata.statistics.Statistics;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+
+/**
+ * @Author: LiuDaWei
+ * @Create: 2020年01月05日
+ */
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365779022
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
 
 Review comment:
   ```suggestion
     private void unpackOverlappedFiles() throws IOException {
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365787811
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > unSeqStartTime) {
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365649321
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/aggregation/AggregateFunction.java
 ##########
 @@ -46,6 +47,13 @@ public AggregateFunction(TSDataType dataType) {
 
   public abstract AggreResultData getResult();
 
+
+  public abstract void calculateValueFromStatistics(Statistics chunkStatistics)
+      throws QueryProcessException;
+
+  public abstract void calculateValueFromPageData(BatchData dataInThisPage) throws IOException;
+
+  public abstract void calculateValueFromPageData(BatchData dataInThisPage,long bound) throws IOException;
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365576719
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/universal/PriorityMergeReader.java
 ##########
 @@ -32,19 +32,20 @@
   PriorityQueue<Element> heap = new PriorityQueue<>((o1, o2) -> {
     int timeCompare = Long.compare(o1.timeValuePair.getTimestamp(),
         o2.timeValuePair.getTimestamp());
-    return timeCompare != 0 ? timeCompare : Integer.compare(o2.priority, o1.priority);
+    return timeCompare != 0 ? timeCompare : Long.compare(o2.priority, o1.priority);
 
 Review comment:
   The type of priority is changed to `long` just because in `AbstractDataReader`, `metaData.getVersion()`  is adapted to be the   priority of ChunkReader... Do you think it is necessary?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365669034
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/aggregation/impl/FirstValueAggrFunc.java
 ##########
 @@ -45,6 +46,42 @@ public AggreResultData getResult() {
     return resultData;
   }
 
+  @Override
+  public void calculateValueFromStatistics(Statistics chunkStatistics)
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365579440
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.DiskChunkReader;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      this.filter = filter;
+    }
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  protected boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new DiskChunkReader(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader.addReaderWithPriority(new DiskChunkReader(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  protected BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    return null;
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   *
+   * @param seriesPath
+   * @param filter
+   */
+  private void removeInvalidFiles(Path seriesPath, Filter filter) {
 
 Review comment:
   Param `Path seriesPath` of this method is not necessary because the global variable `seriesPath` can be used. What do you think?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] jixuan1989 commented on issue #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

jixuan1989 commented on issue #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#issuecomment-573270613
 
 
   please try to merge master to check whether the failed test can pass.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365642416
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/fill/LinearFill.java
 ##########
 @@ -80,8 +83,12 @@ public void constructReaders(Path path, QueryContext context)
   public IPointReader getFillResult() throws IOException, UnSupportedFillTypeException {
     TimeValuePair beforePair = null;
     TimeValuePair afterPair = null;
-    while (allDataReader.hasNext()) {
-      afterPair = allDataReader.next();
+    while (batchData.hasCurrent() || allDataReader.hasNextBatch()) {
+      if (!batchData.hasCurrent()) {
+        batchData = allDataReader.nextBatch();
+      }
+      afterPair = new TimeValuePair(batchData.currentTime(), batchData.currentTsPrimitiveType());
 
 Review comment:
   could we assure each batch from RawDataReader is not empty?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365786255
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > unSeqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+    while (!seqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = seqFileResource.get(0).getStartTimeMap();
+      Long seqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > seqStartTime) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * Because seq data and unseq data intersect, the minimum startTime taken from two files at a time
+   * is used as the reference time to start reading data
+   */
+  private void searchMinimumChunk() {
+    hasCachedNextChunk = true;
+    if (!seqChunkMetadatas.isEmpty() && unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = seqChunkMetadatas.remove(0);
+    } else if (seqChunkMetadatas.isEmpty() && !unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = unseqChunkMetadatas.pollFirst();
+    } else if (!seqChunkMetadatas.isEmpty()) {
+      // seq 和 unseq 的 chunk metadata 都不为空
+      if (seqChunkMetadatas.get(0).getStartTime() <= unseqChunkMetadatas.first().getStartTime()) {
+        chunkMetaData = seqChunkMetadatas.remove(0);
+      } else {
+        chunkMetaData = unseqChunkMetadatas.pollFirst();
+      }
+    } else {
+      hasCachedNextChunk = false;
+    }
+  }
+
+  /**
+   * Before reading the chunks, should first clean up all the useless chunks, because in the file
+   * hierarchy, although the files are available, some of the internal chunks are still unavailable
+   *
+   * @throws IOException
+   */
+  private void removeInvalidChunks() throws IOException {
+    //remove seq chunks
+    while (filter != null && (!seqChunkMetadatas.isEmpty() || !seqFileResource.isEmpty())) {
+      while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+      }
+
+      ChunkMetaData metaData = seqChunkMetadatas.get(0);
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        seqChunkMetadatas.remove(0);
+        continue;
+      }
+      break;
+    }
+    while (filter != null && (!unseqChunkMetadatas.isEmpty() || !unseqFileResource.isEmpty())) {
+      while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+      }
+      ChunkMetaData metaData = unseqChunkMetadatas.first();
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        unseqChunkMetadatas.pollFirst();
+        continue;
+      }
+      break;
+    }
+    fillMetadataContainer();
+  }
+
+  /**
+   * Because there may be too many files in the scenario used by the user, we cannot open all the
+   * chunks at once, which may OOM, so we can only fill one file at a time when needed. This
+   * approach is likely to be ubiquitous, but it keeps the system running smoothly
+   *
+   * @throws IOException
+   */
+  private void fillMetadataContainer() throws IOException {
+    while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+      seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+    }
+    while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+      unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+    }
+  }
+
+  /**
+   * Before calling this method, you should make sure that all the intersecting files are filled in
+   * the container, because the files intersect, but some chunks may still be useless, so you need
+   * to clean up all the unused chunks and populate the container. It should be noted that this
+   * chunk collection is not in order, and all chunks should be used at once
+   */
+  private void fillOverlappedChunks() throws IOException {
+    while (!unseqChunkMetadatas.isEmpty()) {
+      long startTime = unseqChunkMetadatas.first().getStartTime();
+
+      if (chunkMetaData.getEndTime() > startTime) {
+        ChunkMetaData metaData = unseqChunkMetadatas.pollFirst();
+        IChunkReader chunkReader = initChunkReader(metaData);
+        //When data points overlap, there should be a weight
+        overlappedChunkReader.add(new VersionPair<>(metaData.getVersion(), chunkReader));
+        continue;
+      }
+      break;
+    }
+    while (!seqChunkMetadatas.isEmpty()) {
+      long startTime = seqChunkMetadatas.get(0).getStartTime();
+
+      if (chunkMetaData.getEndTime() > startTime) {
 
 Review comment:
   ```suggestion
         if (chunkMetaData.getEndTime() >= startTime) {
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365649269
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.DiskChunkReader;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      this.filter = filter;
+    }
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  protected boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new DiskChunkReader(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader.addReaderWithPriority(new DiskChunkReader(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  protected BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    return null;
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   *
+   * @param seriesPath
+   * @param filter
+   */
+  private void removeInvalidFiles(Path seriesPath, Filter filter) {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      TsFileResource tsFileResource = seqFileResource.get(0);
+      long startTime = tsFileResource.getStartTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      long endTime = tsFileResource.getEndTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+      if (!filter.satisfyStartEndTime(startTime, endTime)) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365664307
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkReaderIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
 
 Review comment:
   what if filter == null and has a TTL?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365554360
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/engine/querycontext/ReadOnlyMemChunk.java
 ##########
 @@ -107,4 +112,40 @@ public boolean isEmpty() {
     checkInitialized();
     return sortedTimeValuePairList.isEmpty();
   }
+
+  public ChunkMetaData getChunkMetaData() {
+    Statistics statsByType = Statistics.getStatsByType(dataType);
+    ChunkMetaData metaData = new ChunkMetaData(measurementUid, dataType, 0, statsByType);
+    if (!isEmpty()) {
+      List<TimeValuePair> sortedTimeValuePairList = getSortedTimeValuePairList();
 
 Review comment:
   could TimeValuePairSorter be replaced by TVList?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365670411
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/executor/AggregateEngineExecutor.java
 ##########
 @@ -1,336 +1,324 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.iotdb.db.query.executor;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-import org.apache.iotdb.db.conf.IoTDBDescriptor;
-import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
-import org.apache.iotdb.db.exception.StorageEngineException;
-import org.apache.iotdb.db.exception.path.PathException;
-import org.apache.iotdb.db.exception.query.QueryProcessException;
-import org.apache.iotdb.db.qp.physical.crud.AggregationPlan;
-import org.apache.iotdb.db.query.aggregation.AggreResultData;
-import org.apache.iotdb.db.query.aggregation.AggregateFunction;
-import org.apache.iotdb.db.query.aggregation.impl.LastValueAggrFunc;
-import org.apache.iotdb.db.query.aggregation.impl.MaxTimeAggrFunc;
-import org.apache.iotdb.db.query.context.QueryContext;
-import org.apache.iotdb.db.query.control.QueryResourceManager;
-import org.apache.iotdb.db.query.dataset.AggreResultDataPointReader;
-import org.apache.iotdb.db.query.dataset.OldEngineDataSetWithoutValueFilter;
-import org.apache.iotdb.db.query.factory.AggreFuncFactory;
-import org.apache.iotdb.db.query.reader.IPointReader;
-import org.apache.iotdb.db.query.reader.IReaderByTimestamp;
-import org.apache.iotdb.db.query.reader.resourceRelated.OldUnseqResourceMergeReader;
-import org.apache.iotdb.db.query.reader.resourceRelated.SeqResourceIterateReader;
-import org.apache.iotdb.db.query.reader.seriesRelated.SeriesReaderByTimestamp;
-import org.apache.iotdb.db.query.timegenerator.EngineTimeGenerator;
-import org.apache.iotdb.tsfile.file.header.PageHeader;
-import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
-import org.apache.iotdb.tsfile.read.common.BatchData;
-import org.apache.iotdb.tsfile.read.common.Path;
-import org.apache.iotdb.tsfile.read.expression.IExpression;
-import org.apache.iotdb.tsfile.read.expression.impl.GlobalTimeExpression;
-import org.apache.iotdb.tsfile.read.filter.basic.Filter;
-import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
-import org.apache.iotdb.tsfile.read.reader.IAggregateReader;
-
-public class AggregateEngineExecutor {
-
-  private List<Path> selectedSeries;
-  private List<TSDataType> dataTypes;
-  private List<String> aggres;
-  private IExpression expression;
-
-  /**
-   * aggregation batch calculation size.
-   **/
-  private int aggregateFetchSize;
-
-  /**
-   * constructor.
-   */
-  public AggregateEngineExecutor(AggregationPlan aggregationPlan) {
-    this.selectedSeries = aggregationPlan.getDeduplicatedPaths();
-    this.dataTypes = aggregationPlan.getDeduplicatedDataTypes();
-    this.aggres = aggregationPlan.getDeduplicatedAggregations();
-    this.expression = aggregationPlan.getExpression();
-    this.aggregateFetchSize = IoTDBDescriptor.getInstance().getConfig().getBatchSize();
-  }
-
-  /**
-   * execute aggregate function with only time filter or no filter.
-   *
-   * @param context query context
-   */
-  public QueryDataSet executeWithoutValueFilter(QueryContext context)
-      throws StorageEngineException, IOException, QueryProcessException {
-    Filter timeFilter = null;
-    if (expression != null) {
-      timeFilter = ((GlobalTimeExpression) expression).getFilter();
-    }
-
-    List<IAggregateReader> readersOfSequenceData = new ArrayList<>();
-    List<IPointReader> readersOfUnSequenceData = new ArrayList<>();
-    List<AggregateFunction> aggregateFunctions = new ArrayList<>();
-    for (int i = 0; i < selectedSeries.size(); i++) {
-      // construct AggregateFunction
-      TSDataType tsDataType = dataTypes.get(i);
-      AggregateFunction function = AggreFuncFactory.getAggrFuncByName(aggres.get(i), tsDataType);
-      function.init();
-      aggregateFunctions.add(function);
-
-      QueryDataSource queryDataSource = QueryResourceManager.getInstance()
-          .getQueryDataSource(selectedSeries.get(i), context);
-      // add additional time filter if TTL is set
-      timeFilter = queryDataSource.setTTL(timeFilter);
-
-      // sequence reader for sealed tsfile, unsealed tsfile, memory
-      IAggregateReader seqResourceIterateReader;
-      if (function instanceof MaxTimeAggrFunc || function instanceof LastValueAggrFunc) {
-        seqResourceIterateReader = new SeqResourceIterateReader(queryDataSource.getSeriesPath(),
-            queryDataSource.getSeqResources(), timeFilter, context, true);
-      } else {
-        seqResourceIterateReader = new SeqResourceIterateReader(queryDataSource.getSeriesPath(),
-            queryDataSource.getSeqResources(), timeFilter, context, false);
-      }
-
-      // unseq reader for all chunk groups in unSeqFile, memory
-      IPointReader unseqResourceMergeReader = new OldUnseqResourceMergeReader(
-          queryDataSource.getSeriesPath(),
-          queryDataSource.getUnseqResources(), context, timeFilter);
-
-      readersOfSequenceData.add(seqResourceIterateReader);
-      readersOfUnSequenceData.add(unseqResourceMergeReader);
-    }
-    List<AggreResultData> aggreResultDataList = new ArrayList<>();
-    //TODO use multi-thread
-    for (int i = 0; i < selectedSeries.size(); i++) {
-      AggreResultData aggreResultData = aggregateWithoutValueFilter(aggregateFunctions.get(i),
-          readersOfSequenceData.get(i), readersOfUnSequenceData.get(i), timeFilter);
-      aggreResultDataList.add(aggreResultData);
-    }
-    return constructDataSet(aggreResultDataList);
-  }
-
-  /**
-   * calculation aggregate result with only time filter or no filter for one series.
-   *
-   * @param function         aggregate function
-   * @param sequenceReader   sequence data reader
-   * @param unSequenceReader unsequence data reader
-   * @param filter           time filter or null
-   * @return one series aggregate result data
-   */
-  private AggreResultData aggregateWithoutValueFilter(AggregateFunction function,
-      IAggregateReader sequenceReader, IPointReader unSequenceReader, Filter filter)
-      throws IOException, QueryProcessException {
-    if (function instanceof MaxTimeAggrFunc || function instanceof LastValueAggrFunc) {
-      return handleLastMaxTimeWithOutTimeGenerator(function, sequenceReader, unSequenceReader,
-          filter);
-    }
-
-    while (sequenceReader.hasNextBatch()) {
-      PageHeader pageHeader = sequenceReader.nextPageHeader();
-      // judge if overlap with unsequence data
-      if (canUseHeader(function, pageHeader, unSequenceReader, filter)) {
-        // cal by pageHeader
-        function.calculateValueFromPageHeader(pageHeader);
-        sequenceReader.skipPageData();
-      } else {
-        // cal by pageData
-        function.calculateValueFromPageData(sequenceReader.nextBatch(), unSequenceReader);
-      }
-
-      if (function.isCalculatedAggregationResult()) {
-        return function.getResult();
-      }
-    }
-
-    // cal with unsequence data
-    if (unSequenceReader.hasNext()) {
-      function.calculateValueFromUnsequenceReader(unSequenceReader);
-    }
-    return function.getResult();
-  }
-
-  /**
-   * determine whether pageHeader can be used to compute aggregation results.
-   */
-  private boolean canUseHeader(AggregateFunction function, PageHeader pageHeader,
-      IPointReader unSequenceReader, Filter filter)
-      throws IOException, QueryProcessException {
-    // if page data is memory data.
-    if (pageHeader == null) {
-      return false;
-    }
-
-    long minTime = pageHeader.getStartTime();
-    long maxTime = pageHeader.getEndTime();
-
-    // If there are points in the page that do not satisfy the time filter,
-    // page header cannot be used to calculate.
-    if (filter != null && !filter.containStartEndTime(minTime, maxTime)) {
-      return false;
-    }
-
-    // cal unsequence data with timestamps between pages.
-    function.calculateValueFromUnsequenceReader(unSequenceReader, minTime);
-
-    return !(unSequenceReader.hasNext() && unSequenceReader.current().getTimestamp() <= maxTime);
-
-  }
-
-  /**
-   * handle last and max_time aggregate function with only time filter or no filter.
-   *
-   * @param function         aggregate function
-   * @param sequenceReader   sequence data reader
-   * @param unSequenceReader unsequence data reader
-   * @return BatchData-aggregate result
-   */
-  private AggreResultData handleLastMaxTimeWithOutTimeGenerator(AggregateFunction function,
-      IAggregateReader sequenceReader, IPointReader unSequenceReader, Filter timeFilter)
-      throws IOException, QueryProcessException {
-    long lastBatchTimeStamp = Long.MIN_VALUE;
-    boolean isChunkEnd = false;
-    while (sequenceReader.hasNextBatch()) {
-      PageHeader pageHeader = sequenceReader.nextPageHeader();
-      // judge if overlap with unsequence data
-      if (canUseHeader(function, pageHeader, unSequenceReader, timeFilter)) {
-        // cal by pageHeader
-        function.calculateValueFromPageHeader(pageHeader);
-        sequenceReader.skipPageData();
-
-        if (lastBatchTimeStamp > pageHeader.getStartTime()) {
-          // the chunk is end.
-          isChunkEnd = true;
-        } else {
-          // current page and last page are in the same chunk.
-          lastBatchTimeStamp = pageHeader.getStartTime();
-        }
-      } else {
-        // cal by pageData
-        BatchData batchData = sequenceReader.nextBatch();
-        if (batchData.length() > 0) {
-          if (lastBatchTimeStamp > batchData.currentTime()) {
-            // the chunk is end.
-            isChunkEnd = true;
-          } else {
-            // current page and last page are in the same chunk.
-            lastBatchTimeStamp = batchData.currentTime();
-          }
-          function.calculateValueFromPageData(batchData, unSequenceReader);
-        }
-      }
-      if (isChunkEnd) {
-        break;
-      }
-    }
-
-    // cal with unsequence data
-    if (unSequenceReader.hasNext()) {
-      function.calculateValueFromUnsequenceReader(unSequenceReader);
-    }
-    return function.getResult();
-  }
-
-
-  /**
-   * execute aggregate function with value filter.
-   *
-   * @param context query context.
-   */
-  public QueryDataSet executeWithValueFilter(QueryContext context)
-      throws StorageEngineException, PathException, IOException {
-
-    EngineTimeGenerator timestampGenerator = new EngineTimeGenerator(expression, context);
-    List<IReaderByTimestamp> readersOfSelectedSeries = new ArrayList<>();
-    for (Path path : selectedSeries) {
-      SeriesReaderByTimestamp seriesReaderByTimestamp = new SeriesReaderByTimestamp(path, context);
-      readersOfSelectedSeries.add(seriesReaderByTimestamp);
-    }
-
-    List<AggregateFunction> aggregateFunctions = new ArrayList<>();
-    for (int i = 0; i < selectedSeries.size(); i++) {
-      TSDataType type = dataTypes.get(i);
-      AggregateFunction function = AggreFuncFactory.getAggrFuncByName(aggres.get(i), type);
-      function.init();
-      aggregateFunctions.add(function);
-    }
-    List<AggreResultData> batchDataList = aggregateWithValueFilter(aggregateFunctions,
-        timestampGenerator,
-        readersOfSelectedSeries);
-    return constructDataSet(batchDataList);
-  }
-
-  /**
-   * calculation aggregate result with value filter.
-   */
-  private List<AggreResultData> aggregateWithValueFilter(
-      List<AggregateFunction> aggregateFunctions,
-      EngineTimeGenerator timestampGenerator,
-      List<IReaderByTimestamp> readersOfSelectedSeries)
-      throws IOException {
-
-    while (timestampGenerator.hasNext()) {
-
-      // generate timestamps for aggregate
-      long[] timeArray = new long[aggregateFetchSize];
-      int timeArrayLength = 0;
-      for (int cnt = 0; cnt < aggregateFetchSize; cnt++) {
-        if (!timestampGenerator.hasNext()) {
-          break;
-        }
-        timeArray[timeArrayLength++] = timestampGenerator.next();
-      }
-
-      // cal part of aggregate result
-      for (int i = 0; i < readersOfSelectedSeries.size(); i++) {
-        aggregateFunctions.get(i).calcAggregationUsingTimestamps(timeArray, timeArrayLength,
-            readersOfSelectedSeries.get(i));
-      }
-    }
-
-    List<AggreResultData> aggreResultDataArrayList = new ArrayList<>();
-    for (AggregateFunction function : aggregateFunctions) {
-      aggreResultDataArrayList.add(function.getResult());
-    }
-    return aggreResultDataArrayList;
-  }
-
-  /**
-   * using aggregate result data list construct QueryDataSet.
-   *
-   * @param aggreResultDataList aggregate result data list
-   */
-  private QueryDataSet constructDataSet(List<AggreResultData> aggreResultDataList)
-      throws IOException {
-    List<TSDataType> dataTypes = new ArrayList<>();
-    List<IPointReader> resultDataPointReaders = new ArrayList<>();
-    for (AggreResultData resultData : aggreResultDataList) {
-      dataTypes.add(resultData.getDataType());
-      resultDataPointReaders.add(new AggreResultDataPointReader(resultData));
-    }
-    return new OldEngineDataSetWithoutValueFilter(selectedSeries, dataTypes,
-        resultDataPointReaders);
-  }
-}
+///*
+// * Licensed to the Apache Software Foundation (ASF) under one
+// * or more contributor license agreements.  See the NOTICE file
+// * distributed with this work for additional information
+// * regarding copyright ownership.  The ASF licenses this file
+// * to you under the Apache License, Version 2.0 (the
+// * "License"); you may not use this file except in compliance
+// * with the License.  You may obtain a copy of the License at
+// *
+// *     http://www.apache.org/licenses/LICENSE-2.0
+// *
+// * Unless required by applicable law or agreed to in writing,
+// * software distributed under the License is distributed on an
+// * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// * KIND, either express or implied.  See the License for the
+// * specific language governing permissions and limitations
+// * under the License.
+// */
+//
+//package org.apache.iotdb.db.query.executor;
+//
+//import java.io.IOException;
+//import java.util.ArrayList;
+//import java.util.List;
+//import org.apache.iotdb.db.conf.IoTDBDescriptor;
+//import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+//import org.apache.iotdb.db.exception.StorageEngineException;
+//import org.apache.iotdb.db.exception.path.PathException;
+//import org.apache.iotdb.db.exception.query.QueryProcessException;
+//import org.apache.iotdb.db.qp.physical.crud.AggregationPlan;
+//import org.apache.iotdb.db.query.aggregation.AggreResultData;
+//import org.apache.iotdb.db.query.aggregation.AggregateFunction;
+//import org.apache.iotdb.db.query.aggregation.impl.LastValueAggrFunc;
+//import org.apache.iotdb.db.query.aggregation.impl.MaxTimeAggrFunc;
+//import org.apache.iotdb.db.query.context.QueryContext;
+//import org.apache.iotdb.db.query.control.QueryResourceManager;
+//import org.apache.iotdb.db.query.dataset.AggreResultDataPointReader;
+//import org.apache.iotdb.db.query.dataset.OldEngineDataSetWithoutValueFilter;
+//import org.apache.iotdb.db.query.factory.AggreFuncFactory;
+//import org.apache.iotdb.db.query.reader.IPointReader;
+//import org.apache.iotdb.db.query.reader.IReaderByTimestamp;
+//import org.apache.iotdb.db.query.reader.resourceRelated.OldUnseqResourceMergeReader;
+//import org.apache.iotdb.db.query.reader.seriesRelated.NewSeriesReaderWithoutValueFilter;
+//import org.apache.iotdb.db.query.reader.seriesRelated.SeriesReaderByTimestamp;
+//import org.apache.iotdb.db.query.timegenerator.EngineTimeGenerator;
+//import org.apache.iotdb.tsfile.file.header.PageHeader;
+//import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+//import org.apache.iotdb.tsfile.read.common.BatchData;
+//import org.apache.iotdb.tsfile.read.common.Path;
+//import org.apache.iotdb.tsfile.read.expression.IExpression;
+//import org.apache.iotdb.tsfile.read.expression.impl.GlobalTimeExpression;
+//import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+//import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+//import org.apache.iotdb.tsfile.read.reader.IAggregateReader;
+//
+//public class AggregateEngineExecutor {
+//
+//  private List<Path> selectedSeries;
+//  private List<TSDataType> dataTypes;
+//  private List<String> aggres;
+//  private IExpression expression;
+//
+//  /**
+//   * aggregation batch calculation size.
+//   **/
+//  private int aggregateFetchSize;
+//
+//  /**
+//   * constructor.
+//   */
+//  public AggregateEngineExecutor(AggregationPlan aggregationPlan) {
 
 Review comment:
   done, and I've removed all the unused classes and tests
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin merged pull request #692: [IOTDB-418] add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin merged pull request #692: [IOTDB-418] add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365649023
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/chunkRelated/DiskChunkReader.java
 ##########
 @@ -36,10 +37,10 @@
  */
 public class DiskChunkReader implements IPointReader, IBatchReader {
 
 Review comment:
   `DiskChunkReader` looks like `Iterator`  ,so i rename `DiskChunkReader` to `ChunkReaderIterator`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365635452
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/dataset/DeviceIterateDataSet.java
 ##########
 @@ -175,15 +175,15 @@ protected boolean hasNextWithoutConstraint() throws IOException {
             groupByPlan.setSlidingStep(slidingStep);
             groupByPlan.setUnit(unit);
             groupByPlan.setDeduplicatedPaths(executePaths);
-            groupByPlan.setDeduplicatedDataTypes(dataTypes);
+            groupByPlan.setDeduplicatedDataTypes(tsDataTypes);
 
 Review comment:
   is this a bug before?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365575618
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/IRawReader.java
 ##########
 @@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+import java.io.IOException;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+
+/**
+ * @Author: LiuDaWei
+ * @Create: 2020年01月06日
+ */
+public interface IRawReader {
 
 Review comment:
   I think this could be renamed to `IRawDataReader` to be consistent with `RawDataReaderWithoutValueFilter` (or rename `RawDataReaderWithoutValueFilter` to  `RawReaderWithoutValueFilter`... just to keep consistent). What do you think?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365653234
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/chunkRelated/ChunkReaderIterator.java
 ##########
 @@ -34,12 +35,12 @@
  * are used here: <code>ChunkReaderWithoutFilter</code> and <code>ChunkReaderWithFilter</code>.
  * <p>
  */
-public class DiskChunkReader implements IPointReader, IBatchReader {
+public class ChunkReaderIterator implements IPointReader, IBatchReader {
 
 Review comment:
   This name is confusing... it seems could iterate chunk reader

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r361959757
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/NewSeriesReaderWithoutValueFilter.java
 ##########
 @@ -52,22 +61,44 @@
 
   private PageHeader cachedPageHeader;
   private boolean hasCachedNextPage;
-
-  public NewSeriesReaderWithoutValueFilter(Path seriesPath, TSDataType dataType, Filter timeFilter,
-      QueryContext context) throws StorageEngineException, IOException {
-    QueryDataSource queryDataSource = QueryResourceManager.getInstance()
-        .getQueryDataSource(seriesPath, context);
-    timeFilter = queryDataSource.setTTL(timeFilter);
+  private IAggregateReader overlappedChunkReader;
+  private long latestDirectlyOverlappedPageEndTime;
+
+  public NewSeriesReaderWithoutValueFilter(QueryDataSource queryDataSource, TSDataType dataType,
+      Filter timeFilter, QueryContext context) throws IOException {
+    Path seriesPath = queryDataSource.getSeriesPath();
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      String queryMeasurement = seriesPath.getMeasurement();
+      List<Long> o1StartTimeList = o1.getChunkMetaDataList().stream()
 
 Review comment:
   This ChunkMetadataList should all belongs to this measurement. No need to filter again. Besides, only un closed TsFile has the chunkMetaDataList.
   
   The sort could compare by the startTimeMap.get(seriesPath.getDeviceId())

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365580443
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/aggregation/AggregateFunction.java
 ##########
 @@ -46,6 +47,13 @@ public AggregateFunction(TSDataType dataType) {
 
   public abstract AggreResultData getResult();
 
+
+  public abstract void calculateValueFromStatistics(Statistics chunkStatistics)
+      throws QueryProcessException;
+
+  public abstract void calculateValueFromPageData(BatchData dataInThisPage) throws IOException;
+
+  public abstract void calculateValueFromPageData(BatchData dataInThisPage,long bound) throws IOException;
 
 Review comment:
   Please remember to add java doc for these new added methods 😄

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r361959757
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/NewSeriesReaderWithoutValueFilter.java
 ##########
 @@ -52,22 +61,44 @@
 
   private PageHeader cachedPageHeader;
   private boolean hasCachedNextPage;
-
-  public NewSeriesReaderWithoutValueFilter(Path seriesPath, TSDataType dataType, Filter timeFilter,
-      QueryContext context) throws StorageEngineException, IOException {
-    QueryDataSource queryDataSource = QueryResourceManager.getInstance()
-        .getQueryDataSource(seriesPath, context);
-    timeFilter = queryDataSource.setTTL(timeFilter);
+  private IAggregateReader overlappedChunkReader;
+  private long latestDirectlyOverlappedPageEndTime;
+
+  public NewSeriesReaderWithoutValueFilter(QueryDataSource queryDataSource, TSDataType dataType,
+      Filter timeFilter, QueryContext context) throws IOException {
+    Path seriesPath = queryDataSource.getSeriesPath();
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      String queryMeasurement = seriesPath.getMeasurement();
+      List<Long> o1StartTimeList = o1.getChunkMetaDataList().stream()
 
 Review comment:
   This ChunkMetadataList should all belongs to this measurement. No need to filter again.
   
   The sort could compare by the startTimeMap

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365772430
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
 
 Review comment:
    unseq files are very special files that intersect not only with sequence files, but also with
    other unseq files. So we need to find all tsfiles that overlapped with current chunk and 
    extract chunks from the resource
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r361959757
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/NewSeriesReaderWithoutValueFilter.java
 ##########
 @@ -52,22 +61,44 @@
 
   private PageHeader cachedPageHeader;
   private boolean hasCachedNextPage;
-
-  public NewSeriesReaderWithoutValueFilter(Path seriesPath, TSDataType dataType, Filter timeFilter,
-      QueryContext context) throws StorageEngineException, IOException {
-    QueryDataSource queryDataSource = QueryResourceManager.getInstance()
-        .getQueryDataSource(seriesPath, context);
-    timeFilter = queryDataSource.setTTL(timeFilter);
+  private IAggregateReader overlappedChunkReader;
+  private long latestDirectlyOverlappedPageEndTime;
+
+  public NewSeriesReaderWithoutValueFilter(QueryDataSource queryDataSource, TSDataType dataType,
+      Filter timeFilter, QueryContext context) throws IOException {
+    Path seriesPath = queryDataSource.getSeriesPath();
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      String queryMeasurement = seriesPath.getMeasurement();
+      List<Long> o1StartTimeList = o1.getChunkMetaDataList().stream()
 
 Review comment:
   This ChunkMetadataList should all belongs to this measurement. No need to filter again. Besides, only un closed TsFile has the chunkMetaDataList.
   
   The sort could compare by the startTimeMap

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365649346
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/aggregation/impl/FirstValueAggrFunc.java
 ##########
 @@ -45,6 +46,42 @@ public AggreResultData getResult() {
     return resultData;
   }
 
+  @Override
+  public void calculateValueFromStatistics(Statistics chunkStatistics)
+      throws QueryProcessException {
+    if (resultData.isSetTime()) {
+      return;
+    }
+
+    Object firstVal = chunkStatistics.getFirstValue();
+    if (firstVal == null) {
+      throw new QueryProcessException("ChunkMetaData contains no FIRST value");
+    }
+    resultData.putTimeAndValue(0, firstVal);
+  }
+
+  @Override
+  public void calculateValueFromPageData(BatchData dataInThisPage) throws IOException {
+    if (resultData.isSetTime()) {
+      return;
+    }
+    if (dataInThisPage.hasCurrent()) {
+      resultData.putTimeAndValue(0, dataInThisPage.currentValue());
+    }
+  }
+
+  @Override
+  public void calculateValueFromPageData(BatchData dataInThisPage, long bound) throws IOException {
+    if (resultData.isSetTime()) {
+      return;
+    }
+    if (dataInThisPage.hasCurrent() && dataInThisPage.currentTime() < bound) {
+      resultData.putTimeAndValue(0, dataInThisPage.currentValue());
+      dataInThisPage.next();
+      return;
 
 Review comment:
   yes ,you are right

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365770745
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > unSeqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+    while (!seqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = seqFileResource.get(0).getStartTimeMap();
+      Long seqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > seqStartTime) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * Because seq data and unseq data intersect, the minimum startTime taken from two files at a time
+   * is used as the reference time to start reading data
+   */
+  private void searchMinimumChunk() {
+    hasCachedNextChunk = true;
+    if (!seqChunkMetadatas.isEmpty() && unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = seqChunkMetadatas.remove(0);
+    } else if (seqChunkMetadatas.isEmpty() && !unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = unseqChunkMetadatas.pollFirst();
+    } else if (!seqChunkMetadatas.isEmpty()) {
+      // seq 和 unseq 的 chunk metadata 都不为空
 
 Review comment:
   ```suggestion
         // neither seqChunkMetadatas nor unseqChunkMetadatas is null
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365768651
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > unSeqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+    while (!seqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = seqFileResource.get(0).getStartTimeMap();
+      Long seqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > seqStartTime) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * Because seq data and unseq data intersect, the minimum startTime taken from two files at a time
+   * is used as the reference time to start reading data
+   */
+  private void searchMinimumChunk() {
+    hasCachedNextChunk = true;
+    if (!seqChunkMetadatas.isEmpty() && unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = seqChunkMetadatas.remove(0);
+    } else if (seqChunkMetadatas.isEmpty() && !unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = unseqChunkMetadatas.pollFirst();
+    } else if (!seqChunkMetadatas.isEmpty()) {
+      // seq 和 unseq 的 chunk metadata 都不为空
+      if (seqChunkMetadatas.get(0).getStartTime() <= unseqChunkMetadatas.first().getStartTime()) {
+        chunkMetaData = seqChunkMetadatas.remove(0);
+      } else {
+        chunkMetaData = unseqChunkMetadatas.pollFirst();
+      }
+    } else {
+      hasCachedNextChunk = false;
+    }
+  }
+
+  /**
+   * Before reading the chunks, should first clean up all the useless chunks, because in the file
+   * hierarchy, although the files are available, some of the internal chunks are still unavailable
+   *
+   * @throws IOException
+   */
+  private void removeInvalidChunks() throws IOException {
+    //remove seq chunks
+    while (filter != null && (!seqChunkMetadatas.isEmpty() || !seqFileResource.isEmpty())) {
+      while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+      }
+
+      ChunkMetaData metaData = seqChunkMetadatas.get(0);
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        seqChunkMetadatas.remove(0);
+        continue;
+      }
+      break;
+    }
+    while (filter != null && (!unseqChunkMetadatas.isEmpty() || !unseqFileResource.isEmpty())) {
+      while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+      }
+      ChunkMetaData metaData = unseqChunkMetadatas.first();
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        unseqChunkMetadatas.pollFirst();
+        continue;
+      }
+      break;
+    }
+    fillMetadataContainer();
 
 Review comment:
   is this necessary?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365672685
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/aggregation/AggregateFunction.java
 ##########
 @@ -46,6 +47,33 @@ public AggregateFunction(TSDataType dataType) {
 
   public abstract AggreResultData getResult();
 
+  /**
+   * Calculate the aggregation using Statistics
+   *
+   * @param statistics chunkStatistics or pageStatistics
+   * @throws QueryProcessException
+   */
+  public abstract void calculateValueFromStatistics(Statistics statistics)
+      throws QueryProcessException;
+
+  /**
+   * Aggregate results cannot be calculated using Statistics directly, using the data in each page
+   *
+   * @param dataInThisPage the data in Page
+   * @throws IOException
+   */
+  public abstract void calculateValueFromPageData(BatchData dataInThisPage) throws IOException;
+
+  /**
+   * Aggregate results cannot be calculated using Statistics directly, using the data in each page
+   *
+   * @param dataInThisPage the data in Page
+   * @param bound          the time upper bounder of data in unsequence data reader
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365648851
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/RawDataReaderWithoutValueFilter.java
 ##########
 @@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+
+/**
+ * @Author: LiuDaWei
+ * @Create: 2020年01月06日
+ */
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365785922
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > unSeqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+    while (!seqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = seqFileResource.get(0).getStartTimeMap();
+      Long seqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > seqStartTime) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * Because seq data and unseq data intersect, the minimum startTime taken from two files at a time
+   * is used as the reference time to start reading data
+   */
+  private void searchMinimumChunk() {
+    hasCachedNextChunk = true;
+    if (!seqChunkMetadatas.isEmpty() && unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = seqChunkMetadatas.remove(0);
+    } else if (seqChunkMetadatas.isEmpty() && !unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = unseqChunkMetadatas.pollFirst();
+    } else if (!seqChunkMetadatas.isEmpty()) {
+      // seq 和 unseq 的 chunk metadata 都不为空
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r363852131
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/NewSeriesReaderWithoutValueFilter.java
 ##########
 @@ -52,22 +61,44 @@
 
   private PageHeader cachedPageHeader;
   private boolean hasCachedNextPage;
-
-  public NewSeriesReaderWithoutValueFilter(Path seriesPath, TSDataType dataType, Filter timeFilter,
-      QueryContext context) throws StorageEngineException, IOException {
-    QueryDataSource queryDataSource = QueryResourceManager.getInstance()
-        .getQueryDataSource(seriesPath, context);
-    timeFilter = queryDataSource.setTTL(timeFilter);
+  private IAggregateReader overlappedChunkReader;
+  private long latestDirectlyOverlappedPageEndTime;
+
+  public NewSeriesReaderWithoutValueFilter(QueryDataSource queryDataSource, TSDataType dataType,
+      Filter timeFilter, QueryContext context) throws IOException {
+    Path seriesPath = queryDataSource.getSeriesPath();
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      String queryMeasurement = seriesPath.getMeasurement();
+      List<Long> o1StartTimeList = o1.getChunkMetaDataList().stream()
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365575434
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.DiskChunkReader;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      this.filter = filter;
+    }
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  protected boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new DiskChunkReader(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader.addReaderWithPriority(new DiskChunkReader(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  protected BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    return null;
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   *
+   * @param seriesPath
+   * @param filter
+   */
+  private void removeInvalidFiles(Path seriesPath, Filter filter) {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      TsFileResource tsFileResource = seqFileResource.get(0);
+      long startTime = tsFileResource.getStartTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      long endTime = tsFileResource.getEndTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+      if (!filter.satisfyStartEndTime(startTime, endTime)) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      TsFileResource tsFileResource = unseqFileResource.first();
+      Long startTime = tsFileResource.getStartTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      Long endTime = tsFileResource.getEndTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+      if (!filter.satisfyStartEndTime(startTime, endTime)) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > unSeqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+    while (!seqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = seqFileResource.get(0).getStartTimeMap();
+      Long seqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > seqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * Because seq data and unseq data intersect, the minimum startTime taken from two files at a time
+   * is used as the reference time to start reading data
+   */
+  private void searchMinimumChunk() {
+    hasCachedNextChunk = true;
+    if (!seqChunkMetadatas.isEmpty() && unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = seqChunkMetadatas.remove(0);
+    } else if (seqChunkMetadatas.isEmpty() && !unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = unseqChunkMetadatas.pollFirst();
+    } else if (!seqChunkMetadatas.isEmpty()) {
+      // seq 和 unseq 的 chunk metadata 都不为空
+      if (seqChunkMetadatas.get(0).getStartTime() <= unseqChunkMetadatas.first().getStartTime()) {
+        chunkMetaData = seqChunkMetadatas.remove(0);
+      } else {
+        chunkMetaData = unseqChunkMetadatas.pollFirst();
+      }
+    } else {
+      hasCachedNextChunk = false;
+    }
+  }
+
+  /**
+   * Before reading the chunks, should first clean up all the useless chunks, because in the file
+   * hierarchy, although the files are available, some of the internal chunks are still unavailable
+   *
+   * @throws IOException
+   */
+  private void removeInvalidChunks() throws IOException {
+    //remove seq chunks
+    while (filter != null && (!seqChunkMetadatas.isEmpty() || !seqFileResource.isEmpty())) {
+      while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+      }
+
+      ChunkMetaData metaData = seqChunkMetadatas.get(0);
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        seqChunkMetadatas.remove(0);
+        continue;
+      }
+      break;
+    }
+    while (filter != null && (!unseqChunkMetadatas.isEmpty() || !unseqFileResource.isEmpty())) {
+      while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+      }
+      ChunkMetaData metaData = unseqChunkMetadatas.first();
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        unseqChunkMetadatas.pollFirst();
+        continue;
+      }
+      break;
+    }
+    fillMetadataContainer();
+  }
+
+  /**
+   * Because there may be too many files in the scenario used by the user, we cannot open all the
+   * chunks at once, which may OOM, so we can only fill one file at a time when needed. This
+   * approach is likely to be ubiquitous, but it keeps the system running smoothly
+   *
+   * @throws IOException
+   */
+  private void fillMetadataContainer() throws IOException {
+    while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+      seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+    }
+    while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+      unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+    }
+  }
+
+  /**
+   * Before calling this method, you should make sure that all the intersecting files are filled in
+   * the container, because the files intersect, but some chunks may still be useless, so you need
+   * to clean up all the unused chunks and populate the container. It should be noted that this
+   * chunk collection is not in order, and all chunks should be used at once
+   */
+  private void fillOverlappedChunks() throws IOException {
+    while (!unseqChunkMetadatas.isEmpty()) {
+      long startTime = unseqChunkMetadatas.first().getStartTime();
+
+      if (chunkMetaData.getEndTime() > startTime) {
+        ChunkMetaData metaData = unseqChunkMetadatas.pollFirst();
+        IChunkReader chunkReader = initChunkReader(metaData);
+        //When data points overlap, there should be a weight
+        overlappedChunkReader.add(new VersionPair<>(metaData.getVersion(), chunkReader));
+        continue;
+      }
+      break;
+    }
+    while (!seqChunkMetadatas.isEmpty()) {
+      long startTime = seqChunkMetadatas.get(0).getStartTime();
+
+      if (chunkMetaData.getEndTime() > startTime) {
+        ChunkMetaData metaData = seqChunkMetadatas.remove(0);
+        IChunkReader chunkReader = initChunkReader(metaData);
+        overlappedChunkReader.add(new VersionPair<>(metaData.getVersion(), chunkReader));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * This is just a fake page container, because no matter how chaotic the situation is, the data in
+   * a chunk is always in order, so when the first page of the chunk cannot be used, there is no
+   * need to look at the whole chunk.
+   */
+  private void fillOverlappedPages() {
+    currentPage = chunkReader.nextPageHeader();
+    hasCachedNextPage = true;
+    latestDirectlyOverlappedPageEndTime = currentPage.getEndTime();
+    while (!overlappedChunkReader.isEmpty()) {
+      VersionPair<IChunkReader> iChunkReader = overlappedChunkReader.get(0);
+      if (currentPage.getEndTime() > iChunkReader.data.nextPageHeader().getStartTime()) {
+        overlappedPages.add(overlappedChunkReader.remove(0));
+      }
+    }
+  }
+
+  private class VersionPair<T> {
+
+    private long version;
+    private T data;
+
+    public VersionPair(long version, T data) {
+      this.version = version;
+      this.data = data;
+    }
+  }
+
+  public void close() throws IOException {
+    if (chunkMetaData != null) {
+      chunkMetaData.getChunkLoader().close();
+    }
+    for (int i = 0; i < openedChunkLoaders.size(); i++) {
+      openedChunkLoaders.get(i).close();
+    }
 
 Review comment:
   could use for-each to avoid variable `i`:
   ```
   for (IChunkLoader openedChunkLoader : openedChunkLoaders) {
         openedChunkLoader.close();
   }
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365579909
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java
 ##########
 @@ -606,7 +606,8 @@ public String getStorageGroupName() {
       }
       // memSeriesLazyMerger has handled the props,
       // so we do not need to handle it again in the following readOnlyMemChunk
-      ReadOnlyMemChunk timeValuePairSorter = new ReadOnlyMemChunk(dataType, memSeriesLazyMerger,
+      ReadOnlyMemChunk timeValuePairSorter = new ReadOnlyMemChunk(measurementId, dataType,
+          memSeriesLazyMerger,
           Collections.emptyMap());
 
 Review comment:
   could be in one line

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365635891
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/executor/AggregateEngineExecutor.java
 ##########
 @@ -1,336 +1,324 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.iotdb.db.query.executor;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-import org.apache.iotdb.db.conf.IoTDBDescriptor;
-import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
-import org.apache.iotdb.db.exception.StorageEngineException;
-import org.apache.iotdb.db.exception.path.PathException;
-import org.apache.iotdb.db.exception.query.QueryProcessException;
-import org.apache.iotdb.db.qp.physical.crud.AggregationPlan;
-import org.apache.iotdb.db.query.aggregation.AggreResultData;
-import org.apache.iotdb.db.query.aggregation.AggregateFunction;
-import org.apache.iotdb.db.query.aggregation.impl.LastValueAggrFunc;
-import org.apache.iotdb.db.query.aggregation.impl.MaxTimeAggrFunc;
-import org.apache.iotdb.db.query.context.QueryContext;
-import org.apache.iotdb.db.query.control.QueryResourceManager;
-import org.apache.iotdb.db.query.dataset.AggreResultDataPointReader;
-import org.apache.iotdb.db.query.dataset.OldEngineDataSetWithoutValueFilter;
-import org.apache.iotdb.db.query.factory.AggreFuncFactory;
-import org.apache.iotdb.db.query.reader.IPointReader;
-import org.apache.iotdb.db.query.reader.IReaderByTimestamp;
-import org.apache.iotdb.db.query.reader.resourceRelated.OldUnseqResourceMergeReader;
-import org.apache.iotdb.db.query.reader.resourceRelated.SeqResourceIterateReader;
-import org.apache.iotdb.db.query.reader.seriesRelated.SeriesReaderByTimestamp;
-import org.apache.iotdb.db.query.timegenerator.EngineTimeGenerator;
-import org.apache.iotdb.tsfile.file.header.PageHeader;
-import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
-import org.apache.iotdb.tsfile.read.common.BatchData;
-import org.apache.iotdb.tsfile.read.common.Path;
-import org.apache.iotdb.tsfile.read.expression.IExpression;
-import org.apache.iotdb.tsfile.read.expression.impl.GlobalTimeExpression;
-import org.apache.iotdb.tsfile.read.filter.basic.Filter;
-import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
-import org.apache.iotdb.tsfile.read.reader.IAggregateReader;
-
-public class AggregateEngineExecutor {
-
-  private List<Path> selectedSeries;
-  private List<TSDataType> dataTypes;
-  private List<String> aggres;
-  private IExpression expression;
-
-  /**
-   * aggregation batch calculation size.
-   **/
-  private int aggregateFetchSize;
-
-  /**
-   * constructor.
-   */
-  public AggregateEngineExecutor(AggregationPlan aggregationPlan) {
-    this.selectedSeries = aggregationPlan.getDeduplicatedPaths();
-    this.dataTypes = aggregationPlan.getDeduplicatedDataTypes();
-    this.aggres = aggregationPlan.getDeduplicatedAggregations();
-    this.expression = aggregationPlan.getExpression();
-    this.aggregateFetchSize = IoTDBDescriptor.getInstance().getConfig().getBatchSize();
-  }
-
-  /**
-   * execute aggregate function with only time filter or no filter.
-   *
-   * @param context query context
-   */
-  public QueryDataSet executeWithoutValueFilter(QueryContext context)
-      throws StorageEngineException, IOException, QueryProcessException {
-    Filter timeFilter = null;
-    if (expression != null) {
-      timeFilter = ((GlobalTimeExpression) expression).getFilter();
-    }
-
-    List<IAggregateReader> readersOfSequenceData = new ArrayList<>();
-    List<IPointReader> readersOfUnSequenceData = new ArrayList<>();
-    List<AggregateFunction> aggregateFunctions = new ArrayList<>();
-    for (int i = 0; i < selectedSeries.size(); i++) {
-      // construct AggregateFunction
-      TSDataType tsDataType = dataTypes.get(i);
-      AggregateFunction function = AggreFuncFactory.getAggrFuncByName(aggres.get(i), tsDataType);
-      function.init();
-      aggregateFunctions.add(function);
-
-      QueryDataSource queryDataSource = QueryResourceManager.getInstance()
-          .getQueryDataSource(selectedSeries.get(i), context);
-      // add additional time filter if TTL is set
-      timeFilter = queryDataSource.setTTL(timeFilter);
-
-      // sequence reader for sealed tsfile, unsealed tsfile, memory
-      IAggregateReader seqResourceIterateReader;
-      if (function instanceof MaxTimeAggrFunc || function instanceof LastValueAggrFunc) {
-        seqResourceIterateReader = new SeqResourceIterateReader(queryDataSource.getSeriesPath(),
-            queryDataSource.getSeqResources(), timeFilter, context, true);
-      } else {
-        seqResourceIterateReader = new SeqResourceIterateReader(queryDataSource.getSeriesPath(),
-            queryDataSource.getSeqResources(), timeFilter, context, false);
-      }
-
-      // unseq reader for all chunk groups in unSeqFile, memory
-      IPointReader unseqResourceMergeReader = new OldUnseqResourceMergeReader(
-          queryDataSource.getSeriesPath(),
-          queryDataSource.getUnseqResources(), context, timeFilter);
-
-      readersOfSequenceData.add(seqResourceIterateReader);
-      readersOfUnSequenceData.add(unseqResourceMergeReader);
-    }
-    List<AggreResultData> aggreResultDataList = new ArrayList<>();
-    //TODO use multi-thread
-    for (int i = 0; i < selectedSeries.size(); i++) {
-      AggreResultData aggreResultData = aggregateWithoutValueFilter(aggregateFunctions.get(i),
-          readersOfSequenceData.get(i), readersOfUnSequenceData.get(i), timeFilter);
-      aggreResultDataList.add(aggreResultData);
-    }
-    return constructDataSet(aggreResultDataList);
-  }
-
-  /**
-   * calculation aggregate result with only time filter or no filter for one series.
-   *
-   * @param function         aggregate function
-   * @param sequenceReader   sequence data reader
-   * @param unSequenceReader unsequence data reader
-   * @param filter           time filter or null
-   * @return one series aggregate result data
-   */
-  private AggreResultData aggregateWithoutValueFilter(AggregateFunction function,
-      IAggregateReader sequenceReader, IPointReader unSequenceReader, Filter filter)
-      throws IOException, QueryProcessException {
-    if (function instanceof MaxTimeAggrFunc || function instanceof LastValueAggrFunc) {
-      return handleLastMaxTimeWithOutTimeGenerator(function, sequenceReader, unSequenceReader,
-          filter);
-    }
-
-    while (sequenceReader.hasNextBatch()) {
-      PageHeader pageHeader = sequenceReader.nextPageHeader();
-      // judge if overlap with unsequence data
-      if (canUseHeader(function, pageHeader, unSequenceReader, filter)) {
-        // cal by pageHeader
-        function.calculateValueFromPageHeader(pageHeader);
-        sequenceReader.skipPageData();
-      } else {
-        // cal by pageData
-        function.calculateValueFromPageData(sequenceReader.nextBatch(), unSequenceReader);
-      }
-
-      if (function.isCalculatedAggregationResult()) {
-        return function.getResult();
-      }
-    }
-
-    // cal with unsequence data
-    if (unSequenceReader.hasNext()) {
-      function.calculateValueFromUnsequenceReader(unSequenceReader);
-    }
-    return function.getResult();
-  }
-
-  /**
-   * determine whether pageHeader can be used to compute aggregation results.
-   */
-  private boolean canUseHeader(AggregateFunction function, PageHeader pageHeader,
-      IPointReader unSequenceReader, Filter filter)
-      throws IOException, QueryProcessException {
-    // if page data is memory data.
-    if (pageHeader == null) {
-      return false;
-    }
-
-    long minTime = pageHeader.getStartTime();
-    long maxTime = pageHeader.getEndTime();
-
-    // If there are points in the page that do not satisfy the time filter,
-    // page header cannot be used to calculate.
-    if (filter != null && !filter.containStartEndTime(minTime, maxTime)) {
-      return false;
-    }
-
-    // cal unsequence data with timestamps between pages.
-    function.calculateValueFromUnsequenceReader(unSequenceReader, minTime);
-
-    return !(unSequenceReader.hasNext() && unSequenceReader.current().getTimestamp() <= maxTime);
-
-  }
-
-  /**
-   * handle last and max_time aggregate function with only time filter or no filter.
-   *
-   * @param function         aggregate function
-   * @param sequenceReader   sequence data reader
-   * @param unSequenceReader unsequence data reader
-   * @return BatchData-aggregate result
-   */
-  private AggreResultData handleLastMaxTimeWithOutTimeGenerator(AggregateFunction function,
-      IAggregateReader sequenceReader, IPointReader unSequenceReader, Filter timeFilter)
-      throws IOException, QueryProcessException {
-    long lastBatchTimeStamp = Long.MIN_VALUE;
-    boolean isChunkEnd = false;
-    while (sequenceReader.hasNextBatch()) {
-      PageHeader pageHeader = sequenceReader.nextPageHeader();
-      // judge if overlap with unsequence data
-      if (canUseHeader(function, pageHeader, unSequenceReader, timeFilter)) {
-        // cal by pageHeader
-        function.calculateValueFromPageHeader(pageHeader);
-        sequenceReader.skipPageData();
-
-        if (lastBatchTimeStamp > pageHeader.getStartTime()) {
-          // the chunk is end.
-          isChunkEnd = true;
-        } else {
-          // current page and last page are in the same chunk.
-          lastBatchTimeStamp = pageHeader.getStartTime();
-        }
-      } else {
-        // cal by pageData
-        BatchData batchData = sequenceReader.nextBatch();
-        if (batchData.length() > 0) {
-          if (lastBatchTimeStamp > batchData.currentTime()) {
-            // the chunk is end.
-            isChunkEnd = true;
-          } else {
-            // current page and last page are in the same chunk.
-            lastBatchTimeStamp = batchData.currentTime();
-          }
-          function.calculateValueFromPageData(batchData, unSequenceReader);
-        }
-      }
-      if (isChunkEnd) {
-        break;
-      }
-    }
-
-    // cal with unsequence data
-    if (unSequenceReader.hasNext()) {
-      function.calculateValueFromUnsequenceReader(unSequenceReader);
-    }
-    return function.getResult();
-  }
-
-
-  /**
-   * execute aggregate function with value filter.
-   *
-   * @param context query context.
-   */
-  public QueryDataSet executeWithValueFilter(QueryContext context)
-      throws StorageEngineException, PathException, IOException {
-
-    EngineTimeGenerator timestampGenerator = new EngineTimeGenerator(expression, context);
-    List<IReaderByTimestamp> readersOfSelectedSeries = new ArrayList<>();
-    for (Path path : selectedSeries) {
-      SeriesReaderByTimestamp seriesReaderByTimestamp = new SeriesReaderByTimestamp(path, context);
-      readersOfSelectedSeries.add(seriesReaderByTimestamp);
-    }
-
-    List<AggregateFunction> aggregateFunctions = new ArrayList<>();
-    for (int i = 0; i < selectedSeries.size(); i++) {
-      TSDataType type = dataTypes.get(i);
-      AggregateFunction function = AggreFuncFactory.getAggrFuncByName(aggres.get(i), type);
-      function.init();
-      aggregateFunctions.add(function);
-    }
-    List<AggreResultData> batchDataList = aggregateWithValueFilter(aggregateFunctions,
-        timestampGenerator,
-        readersOfSelectedSeries);
-    return constructDataSet(batchDataList);
-  }
-
-  /**
-   * calculation aggregate result with value filter.
-   */
-  private List<AggreResultData> aggregateWithValueFilter(
-      List<AggregateFunction> aggregateFunctions,
-      EngineTimeGenerator timestampGenerator,
-      List<IReaderByTimestamp> readersOfSelectedSeries)
-      throws IOException {
-
-    while (timestampGenerator.hasNext()) {
-
-      // generate timestamps for aggregate
-      long[] timeArray = new long[aggregateFetchSize];
-      int timeArrayLength = 0;
-      for (int cnt = 0; cnt < aggregateFetchSize; cnt++) {
-        if (!timestampGenerator.hasNext()) {
-          break;
-        }
-        timeArray[timeArrayLength++] = timestampGenerator.next();
-      }
-
-      // cal part of aggregate result
-      for (int i = 0; i < readersOfSelectedSeries.size(); i++) {
-        aggregateFunctions.get(i).calcAggregationUsingTimestamps(timeArray, timeArrayLength,
-            readersOfSelectedSeries.get(i));
-      }
-    }
-
-    List<AggreResultData> aggreResultDataArrayList = new ArrayList<>();
-    for (AggregateFunction function : aggregateFunctions) {
-      aggreResultDataArrayList.add(function.getResult());
-    }
-    return aggreResultDataArrayList;
-  }
-
-  /**
-   * using aggregate result data list construct QueryDataSet.
-   *
-   * @param aggreResultDataList aggregate result data list
-   */
-  private QueryDataSet constructDataSet(List<AggreResultData> aggreResultDataList)
-      throws IOException {
-    List<TSDataType> dataTypes = new ArrayList<>();
-    List<IPointReader> resultDataPointReaders = new ArrayList<>();
-    for (AggreResultData resultData : aggreResultDataList) {
-      dataTypes.add(resultData.getDataType());
-      resultDataPointReaders.add(new AggreResultDataPointReader(resultData));
-    }
-    return new OldEngineDataSetWithoutValueFilter(selectedSeries, dataTypes,
-        resultDataPointReaders);
-  }
-}
+///*
+// * Licensed to the Apache Software Foundation (ASF) under one
+// * or more contributor license agreements.  See the NOTICE file
+// * distributed with this work for additional information
+// * regarding copyright ownership.  The ASF licenses this file
+// * to you under the Apache License, Version 2.0 (the
+// * "License"); you may not use this file except in compliance
+// * with the License.  You may obtain a copy of the License at
+// *
+// *     http://www.apache.org/licenses/LICENSE-2.0
+// *
+// * Unless required by applicable law or agreed to in writing,
+// * software distributed under the License is distributed on an
+// * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// * KIND, either express or implied.  See the License for the
+// * specific language governing permissions and limitations
+// * under the License.
+// */
+//
+//package org.apache.iotdb.db.query.executor;
+//
+//import java.io.IOException;
+//import java.util.ArrayList;
+//import java.util.List;
+//import org.apache.iotdb.db.conf.IoTDBDescriptor;
+//import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+//import org.apache.iotdb.db.exception.StorageEngineException;
+//import org.apache.iotdb.db.exception.path.PathException;
+//import org.apache.iotdb.db.exception.query.QueryProcessException;
+//import org.apache.iotdb.db.qp.physical.crud.AggregationPlan;
+//import org.apache.iotdb.db.query.aggregation.AggreResultData;
+//import org.apache.iotdb.db.query.aggregation.AggregateFunction;
+//import org.apache.iotdb.db.query.aggregation.impl.LastValueAggrFunc;
+//import org.apache.iotdb.db.query.aggregation.impl.MaxTimeAggrFunc;
+//import org.apache.iotdb.db.query.context.QueryContext;
+//import org.apache.iotdb.db.query.control.QueryResourceManager;
+//import org.apache.iotdb.db.query.dataset.AggreResultDataPointReader;
+//import org.apache.iotdb.db.query.dataset.OldEngineDataSetWithoutValueFilter;
+//import org.apache.iotdb.db.query.factory.AggreFuncFactory;
+//import org.apache.iotdb.db.query.reader.IPointReader;
+//import org.apache.iotdb.db.query.reader.IReaderByTimestamp;
+//import org.apache.iotdb.db.query.reader.resourceRelated.OldUnseqResourceMergeReader;
+//import org.apache.iotdb.db.query.reader.seriesRelated.NewSeriesReaderWithoutValueFilter;
+//import org.apache.iotdb.db.query.reader.seriesRelated.SeriesReaderByTimestamp;
+//import org.apache.iotdb.db.query.timegenerator.EngineTimeGenerator;
+//import org.apache.iotdb.tsfile.file.header.PageHeader;
+//import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+//import org.apache.iotdb.tsfile.read.common.BatchData;
+//import org.apache.iotdb.tsfile.read.common.Path;
+//import org.apache.iotdb.tsfile.read.expression.IExpression;
+//import org.apache.iotdb.tsfile.read.expression.impl.GlobalTimeExpression;
+//import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+//import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+//import org.apache.iotdb.tsfile.read.reader.IAggregateReader;
+//
+//public class AggregateEngineExecutor {
+//
+//  private List<Path> selectedSeries;
+//  private List<TSDataType> dataTypes;
+//  private List<String> aggres;
+//  private IExpression expression;
+//
+//  /**
+//   * aggregation batch calculation size.
+//   **/
+//  private int aggregateFetchSize;
+//
+//  /**
+//   * constructor.
+//   */
+//  public AggregateEngineExecutor(AggregationPlan aggregationPlan) {
 
 Review comment:
   could this class be removed?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365649284
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.DiskChunkReader;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      this.filter = filter;
+    }
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  protected boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new DiskChunkReader(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader.addReaderWithPriority(new DiskChunkReader(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  protected BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    return null;
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   *
+   * @param seriesPath
+   * @param filter
+   */
+  private void removeInvalidFiles(Path seriesPath, Filter filter) {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      TsFileResource tsFileResource = seqFileResource.get(0);
+      long startTime = tsFileResource.getStartTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      long endTime = tsFileResource.getEndTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+      if (!filter.satisfyStartEndTime(startTime, endTime)) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      TsFileResource tsFileResource = unseqFileResource.first();
+      Long startTime = tsFileResource.getStartTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      Long endTime = tsFileResource.getEndTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+      if (!filter.satisfyStartEndTime(startTime, endTime)) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365787752
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365651600
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/aggregation/AggregateFunction.java
 ##########
 @@ -46,6 +47,33 @@ public AggregateFunction(TSDataType dataType) {
 
   public abstract AggreResultData getResult();
 
+  /**
+   * Calculate the aggregation using Statistics
+   *
+   * @param statistics chunkStatistics or pageStatistics
+   * @throws QueryProcessException
+   */
+  public abstract void calculateValueFromStatistics(Statistics statistics)
+      throws QueryProcessException;
+
+  /**
+   * Aggregate results cannot be calculated using Statistics directly, using the data in each page
+   *
+   * @param dataInThisPage the data in Page
+   * @throws IOException
+   */
+  public abstract void calculateValueFromPageData(BatchData dataInThisPage) throws IOException;
+
+  /**
+   * Aggregate results cannot be calculated using Statistics directly, using the data in each page
+   *
+   * @param dataInThisPage the data in Page
+   * @param bound          the time upper bounder of data in unsequence data reader
 
 Review comment:
   calculate points whose time < bound

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365648837
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/SeriesDataReaderWithoutValueFilter.java
 ##########
 @@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+import java.io.IOException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.statistics.Statistics;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+
+/**
+ * @Author: LiuDaWei
+ * @Create: 2020年01月05日
+ */
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365635678
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/dataset/NewEngineDataSetWithoutValueFilter.java
 ##########
 @@ -147,7 +152,7 @@ public NewEngineDataSetWithoutValueFilter(List<Path> paths, List<TSDataType> dat
   private void init() throws InterruptedException {
     timeHeap = new TreeSet<>();
     for (int i = 0; i < seriesReaderWithoutValueFilterList.size(); i++) {
-      ManagedSeriesReader reader = seriesReaderWithoutValueFilterList.get(i);
+      RawDataReaderWithoutValueFilter reader = seriesReaderWithoutValueFilterList.get(i);
 
 Review comment:
   The ManagedSeriesReader is for distribution, we'd better retain it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365565093
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/RawDataReaderWithoutValueFilter.java
 ##########
 @@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+
+/**
+ * @Author: LiuDaWei
+ * @Create: 2020年01月06日
+ */
 
 Review comment:
   Don't forget to remove it

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365772663
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
 
 Review comment:
   ```suggestion
         Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365772430
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
 
 Review comment:
    unseq files are very special files that intersect not only with sequence files, but also with
    other unseq files. So we need to find all tsfiles that overlapped with current chunk
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365642981
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/fill/PreviousFill.java
 ##########
 @@ -58,8 +61,12 @@ public long getBeforeRange() {
   public IPointReader getFillResult() throws IOException {
     TimeValuePair beforePair = null;
     TimeValuePair cachedPair = null;
-    while (allDataReader.hasNext()) {
-      cachedPair = allDataReader.next();
+    while (batchData.hasCurrent() || allDataReader.hasNextBatch()) {
+      if (!batchData.hasCurrent()) {
+        batchData = allDataReader.nextBatch();
+      }
+      cachedPair = new TimeValuePair(batchData.currentTime(), batchData.currentTsPrimitiveType());
 
 Review comment:
   The same question with LinearFill

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365648875
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/SeriesDataReaderWithoutValueFilter.java
 ##########
 @@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+import java.io.IOException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.statistics.Statistics;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+
+/**
+ * @Author: LiuDaWei
+ * @Create: 2020年01月05日
+ */
+public class SeriesDataReaderWithoutValueFilter extends AbstractDataReader implements
+    IAggregateReader {
+
+  public SeriesDataReaderWithoutValueFilter(Path seriesPath, TSDataType dataType, Filter timeFilter,
+      QueryContext context) throws StorageEngineException, IOException {
+    super(seriesPath, dataType, timeFilter, context);
+  }
+
+
+  @Override
+  public boolean hasNextChunk() throws IOException {
+    return super.hasNextChunk();
+  }
+
+  public boolean canUseChunkStatistics() {
+    Statistics statistics = chunkMetaData.getStatistics();
+    return !overlappedChunkReader.isEmpty() && canUseStatistics(statistics);
+  }
+
+  @Override
+  public Statistics currentChunkStatistics() {
+    return chunkMetaData.getStatistics();
+  }
+
+  @Override
+  public void skipChunkData() throws IOException {
+    hasCachedNextChunk = false;
+  }
+
+
+  @Override
+  public boolean hasNextPage() throws IOException {
+    return super.hasNextPage();
+  }
+
+  @Override
+  public boolean canUsePageStatistics() {
+    Statistics pageStatistics = currentPage.getStatistics();
+    return !overlappedPages.isEmpty() && canUseStatistics(pageStatistics);
+  }
+
+  @Override
+  public Statistics currentPageStatistics() {
+    return currentPage.getStatistics();
+  }
+
+  @Override
+  public void skipPageData() throws IOException {
+    hasCachedNextPage = false;
+  }
+
+
+  @Override
+  public boolean hasNextBatch() throws IOException {
+    return super.hasNextBatch();
+  }
+
+  @Override
+  public BatchData nextBatch() throws IOException {
+    return super.nextBatch();
+  }
+
+
+  protected boolean canUseStatistics(Statistics statistics) {
+    if (filter == null || !filter.containStartEndTime(statistics.getStartTime(),
+        statistics.getEndTime())) {
+      return false;
+    }
+    return true;
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365554668
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/aggregation/AggregateFunction.java
 ##########
 @@ -46,6 +47,13 @@ public AggregateFunction(TSDataType dataType) {
 
   public abstract AggreResultData getResult();
 
+
+  public abstract void calculateValueFromStatistics(Statistics chunkStatistics)
+      throws QueryProcessException;
+
+  public abstract void calculateValueFromPageData(BatchData dataInThisPage) throws IOException;
+
+  public abstract void calculateValueFromPageData(BatchData dataInThisPage,long bound) throws IOException;
 
 Review comment:
   add javadoc for bound
   
   remove unused methods in this class

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r366106209
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> seqResources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = seqResources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    unpackOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
 
 Review comment:
   This line is strange, it's better to remove it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365625630
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/aggregation/impl/FirstValueAggrFunc.java
 ##########
 @@ -45,6 +46,42 @@ public AggreResultData getResult() {
     return resultData;
   }
 
+  @Override
+  public void calculateValueFromStatistics(Statistics chunkStatistics)
 
 Review comment:
   ```suggestion
     public void calculateValueFromStatistics(Statistics statistics)
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365649299
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.DiskChunkReader;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      this.filter = filter;
+    }
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  protected boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new DiskChunkReader(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader.addReaderWithPriority(new DiskChunkReader(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  protected BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    return null;
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   *
+   * @param seriesPath
+   * @param filter
+   */
+  private void removeInvalidFiles(Path seriesPath, Filter filter) {
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365635712
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/dataset/groupby/GroupByWithoutValueFilterDataSet.java
 ##########
 @@ -58,7 +54,6 @@ public GroupByWithoutValueFilterDataSet(QueryContext context, GroupByPlan groupB
       throws PathException, IOException, StorageEngineException {
     super(context, groupByPlan);
 
-    this.unSequenceReaderList = new ArrayList<>();
     this.sequenceReaderList = new ArrayList<>();
     this.timeFilter = null;
     this.hasCachedSequenceDataList = new ArrayList<>();
 
 Review comment:
   variable `hasCachedSequenceDataList` is not used any more in this method, you can delete it

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365649224
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/universal/PriorityMergeReader.java
 ##########
 @@ -32,19 +32,20 @@
   PriorityQueue<Element> heap = new PriorityQueue<>((o1, o2) -> {
     int timeCompare = Long.compare(o1.timeValuePair.getTimestamp(),
         o2.timeValuePair.getTimestamp());
-    return timeCompare != 0 ? timeCompare : Integer.compare(o2.priority, o1.priority);
+    return timeCompare != 0 ? timeCompare : Long.compare(o2.priority, o1.priority);
 
 Review comment:
   The version field can be sorted across all files, so I recommend changing it to long because you don't need to do the extra sorting

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365791970
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/aggregation/AggregateFunction.java
 ##########
 @@ -46,6 +47,13 @@ public AggregateFunction(TSDataType dataType) {
 
   public abstract AggreResultData getResult();
 
+
+  public abstract void calculateValueFromStatistics(Statistics chunkStatistics)
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365789160
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > unSeqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+    while (!seqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = seqFileResource.get(0).getStartTimeMap();
+      Long seqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > seqStartTime) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * Because seq data and unseq data intersect, the minimum startTime taken from two files at a time
+   * is used as the reference time to start reading data
+   */
+  private void searchMinimumChunk() {
+    hasCachedNextChunk = true;
+    if (!seqChunkMetadatas.isEmpty() && unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = seqChunkMetadatas.remove(0);
+    } else if (seqChunkMetadatas.isEmpty() && !unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = unseqChunkMetadatas.pollFirst();
+    } else if (!seqChunkMetadatas.isEmpty()) {
+      // seq 和 unseq 的 chunk metadata 都不为空
+      if (seqChunkMetadatas.get(0).getStartTime() <= unseqChunkMetadatas.first().getStartTime()) {
+        chunkMetaData = seqChunkMetadatas.remove(0);
+      } else {
+        chunkMetaData = unseqChunkMetadatas.pollFirst();
+      }
+    } else {
+      hasCachedNextChunk = false;
+    }
+  }
+
+  /**
+   * Before reading the chunks, should first clean up all the useless chunks, because in the file
+   * hierarchy, although the files are available, some of the internal chunks are still unavailable
+   *
+   * @throws IOException
+   */
+  private void removeInvalidChunks() throws IOException {
+    //remove seq chunks
+    while (filter != null && (!seqChunkMetadatas.isEmpty() || !seqFileResource.isEmpty())) {
+      while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+      }
+
+      ChunkMetaData metaData = seqChunkMetadatas.get(0);
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        seqChunkMetadatas.remove(0);
+        continue;
+      }
+      break;
+    }
+    while (filter != null && (!unseqChunkMetadatas.isEmpty() || !unseqFileResource.isEmpty())) {
+      while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+      }
+      ChunkMetaData metaData = unseqChunkMetadatas.first();
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        unseqChunkMetadatas.pollFirst();
+        continue;
+      }
+      break;
+    }
+    fillMetadataContainer();
+  }
+
+  /**
+   * Because there may be too many files in the scenario used by the user, we cannot open all the
+   * chunks at once, which may OOM, so we can only fill one file at a time when needed. This
+   * approach is likely to be ubiquitous, but it keeps the system running smoothly
+   *
+   * @throws IOException
+   */
+  private void fillMetadataContainer() throws IOException {
+    while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+      seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+    }
+    while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+      unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+    }
+  }
+
+  /**
+   * Before calling this method, you should make sure that all the intersecting files are filled in
+   * the container, because the files intersect, but some chunks may still be useless, so you need
+   * to clean up all the unused chunks and populate the container. It should be noted that this
+   * chunk collection is not in order, and all chunks should be used at once
+   */
+  private void fillOverlappedChunks() throws IOException {
+    while (!unseqChunkMetadatas.isEmpty()) {
+      long startTime = unseqChunkMetadatas.first().getStartTime();
+
+      if (chunkMetaData.getEndTime() > startTime) {
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365575379
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -504,5 +508,8 @@ public void close() throws IOException {
     if (chunkMetaData != null) {
       chunkMetaData.getChunkLoader().close();
     }
+    for (int i = 0; i < openedChunkLoaders.size(); i++) {
+      openedChunkLoaders.get(i).close();
+    }
 
 Review comment:
   could use for-each to avoid variable `i`:
   ```
   for (IChunkLoader openedChunkLoader : openedChunkLoaders) {
         openedChunkLoader.close();
   }
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365565115
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/IRawReader.java
 ##########
 @@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+import java.io.IOException;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+
+/**
+ * @Author: LiuDaWei
+ * @Create: 2020年01月06日
+ */
 
 Review comment:
   Don't forget to remove it

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365786173
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > unSeqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+    while (!seqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = seqFileResource.get(0).getStartTimeMap();
+      Long seqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > seqStartTime) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * Because seq data and unseq data intersect, the minimum startTime taken from two files at a time
+   * is used as the reference time to start reading data
+   */
+  private void searchMinimumChunk() {
+    hasCachedNextChunk = true;
+    if (!seqChunkMetadatas.isEmpty() && unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = seqChunkMetadatas.remove(0);
+    } else if (seqChunkMetadatas.isEmpty() && !unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = unseqChunkMetadatas.pollFirst();
+    } else if (!seqChunkMetadatas.isEmpty()) {
+      // seq 和 unseq 的 chunk metadata 都不为空
+      if (seqChunkMetadatas.get(0).getStartTime() <= unseqChunkMetadatas.first().getStartTime()) {
+        chunkMetaData = seqChunkMetadatas.remove(0);
+      } else {
+        chunkMetaData = unseqChunkMetadatas.pollFirst();
+      }
+    } else {
+      hasCachedNextChunk = false;
+    }
+  }
+
+  /**
+   * Before reading the chunks, should first clean up all the useless chunks, because in the file
+   * hierarchy, although the files are available, some of the internal chunks are still unavailable
+   *
+   * @throws IOException
+   */
+  private void removeInvalidChunks() throws IOException {
+    //remove seq chunks
+    while (filter != null && (!seqChunkMetadatas.isEmpty() || !seqFileResource.isEmpty())) {
+      while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+      }
+
+      ChunkMetaData metaData = seqChunkMetadatas.get(0);
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        seqChunkMetadatas.remove(0);
+        continue;
+      }
+      break;
+    }
+    while (filter != null && (!unseqChunkMetadatas.isEmpty() || !unseqFileResource.isEmpty())) {
+      while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+      }
+      ChunkMetaData metaData = unseqChunkMetadatas.first();
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        unseqChunkMetadatas.pollFirst();
+        continue;
+      }
+      break;
+    }
+    fillMetadataContainer();
+  }
+
+  /**
+   * Because there may be too many files in the scenario used by the user, we cannot open all the
+   * chunks at once, which may OOM, so we can only fill one file at a time when needed. This
+   * approach is likely to be ubiquitous, but it keeps the system running smoothly
+   *
+   * @throws IOException
+   */
+  private void fillMetadataContainer() throws IOException {
+    while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+      seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+    }
+    while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+      unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+    }
+  }
+
+  /**
+   * Before calling this method, you should make sure that all the intersecting files are filled in
+   * the container, because the files intersect, but some chunks may still be useless, so you need
+   * to clean up all the unused chunks and populate the container. It should be noted that this
+   * chunk collection is not in order, and all chunks should be used at once
+   */
+  private void fillOverlappedChunks() throws IOException {
+    while (!unseqChunkMetadatas.isEmpty()) {
+      long startTime = unseqChunkMetadatas.first().getStartTime();
+
+      if (chunkMetaData.getEndTime() > startTime) {
 
 Review comment:
   ```suggestion
         if (chunkMetaData.getEndTime() >= startTime) {
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365786253
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365648865
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/IRawReader.java
 ##########
 @@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+import java.io.IOException;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+
+/**
+ * @Author: LiuDaWei
+ * @Create: 2020年01月06日
+ */
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365579391
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.DiskChunkReader;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      this.filter = filter;
+    }
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  protected boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new DiskChunkReader(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader.addReaderWithPriority(new DiskChunkReader(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  protected BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    return null;
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   *
+   * @param seriesPath
+   * @param filter
+   */
+  private void removeInvalidFiles(Path seriesPath, Filter filter) {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      TsFileResource tsFileResource = seqFileResource.get(0);
+      long startTime = tsFileResource.getStartTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      long endTime = tsFileResource.getEndTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+      if (!filter.satisfyStartEndTime(startTime, endTime)) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      TsFileResource tsFileResource = unseqFileResource.first();
+      Long startTime = tsFileResource.getStartTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      Long endTime = tsFileResource.getEndTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+      if (!filter.satisfyStartEndTime(startTime, endTime)) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
 
 Review comment:
   And this part turns to:
   ```
   while (filter != null && !unseqFileResource.isEmpty()) {
         if (!isValid(unseqFileResource.first())) {
           unseqFileResource.pollFirst();
           continue;
         }
         break;
       }
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365644338
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/chunkRelated/DiskChunkReader.java
 ##########
 @@ -36,10 +37,10 @@
  */
 public class DiskChunkReader implements IPointReader, IBatchReader {
 
 Review comment:
   DiskChunkReader may could be replaced by ChunkReader implement IPointReader.
   Maybe ChunkReaderWrap could be replaced by IChunkReader

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365778559
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
 
 Review comment:
   ```suggestion
         Filter filter, QueryContext context, List<TsFileResource> seqResources) throws IOException {
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365654731
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/universal/PriorityMergeReader.java
 ##########
 @@ -32,19 +32,20 @@
   PriorityQueue<Element> heap = new PriorityQueue<>((o1, o2) -> {
     int timeCompare = Long.compare(o1.timeValuePair.getTimestamp(),
         o2.timeValuePair.getTimestamp());
-    return timeCompare != 0 ? timeCompare : Integer.compare(o2.priority, o1.priority);
+    return timeCompare != 0 ? timeCompare : Long.compare(o2.priority, o1.priority);
 
 Review comment:
   Sure.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365669002
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/aggregation/AggregateFunction.java
 ##########
 @@ -46,6 +47,13 @@ public AggregateFunction(TSDataType dataType) {
 
   public abstract AggreResultData getResult();
 
+
+  public abstract void calculateValueFromStatistics(Statistics chunkStatistics)
+      throws QueryProcessException;
+
+  public abstract void calculateValueFromPageData(BatchData dataInThisPage) throws IOException;
+
+  public abstract void calculateValueFromPageData(BatchData dataInThisPage,long bound) throws IOException;
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365565105
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/IAggregateReader.java
 ##########
 @@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+import java.io.IOException;
+import org.apache.iotdb.tsfile.file.metadata.statistics.Statistics;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+
+/**
+ * @Author: LiuDaWei
+ * @Create: 2020年01月05日
+ */
 
 Review comment:
   Don't forget to remove it

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365554599
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/aggregation/AggregateFunction.java
 ##########
 @@ -46,6 +47,13 @@ public AggregateFunction(TSDataType dataType) {
 
   public abstract AggreResultData getResult();
 
+
+  public abstract void calculateValueFromStatistics(Statistics chunkStatistics)
 
 Review comment:
   I suggest extracting AggreResultData out from AggregateFunction, each time calculateValueFromStatistics returns a AggreResultData, and merge outside.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365581629
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/dataset/DeviceIterateDataSet.java
 ##########
 @@ -175,15 +175,15 @@ protected boolean hasNextWithoutConstraint() throws IOException {
             groupByPlan.setSlidingStep(slidingStep);
             groupByPlan.setUnit(unit);
             groupByPlan.setDeduplicatedPaths(executePaths);
-            groupByPlan.setDeduplicatedDataTypes(dataTypes);
+            groupByPlan.setDeduplicatedDataTypes(tsDataTypes);
 
 Review comment:
   Why changed this?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365573898
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/chunkRelated/DiskChunkReader.java
 ##########
 @@ -36,10 +37,10 @@
  */
 public class DiskChunkReader implements IPointReader, IBatchReader {
 
 Review comment:
   In the description of this PR, you said `Based on this interface, there are two implementations, DiskChunkReader and MemChunkReader`, but `DiskChunkReader` actually doesn't implements `IChunkReader` : )

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365675623
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkReaderIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365579374
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.DiskChunkReader;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      this.filter = filter;
+    }
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  protected boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new DiskChunkReader(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader.addReaderWithPriority(new DiskChunkReader(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  protected BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    return null;
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   *
+   * @param seriesPath
+   * @param filter
+   */
+  private void removeInvalidFiles(Path seriesPath, Filter filter) {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      TsFileResource tsFileResource = seqFileResource.get(0);
+      long startTime = tsFileResource.getStartTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      long endTime = tsFileResource.getEndTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+      if (!filter.satisfyStartEndTime(startTime, endTime)) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
 
 Review comment:
   A new method could be extracted from this method:
   ```
   private boolean isValid(TsFileResource tsFileResource) {
       long startTime = tsFileResource.getStartTimeMap()
           .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
       long endTime = tsFileResource.getEndTimeMap()
           .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
       return filter.satisfyStartEndTime(startTime, endTime);
     }
   ```
   Then this part could be:
   ```
   while (filter != null && !seqFileResource.isEmpty()) {
         if (!isValid(seqFileResource.get(0))) {
           seqFileResource.remove(0);
           continue;
         }
         break;
       }
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365649538
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/dataset/DeviceIterateDataSet.java
 ##########
 @@ -175,15 +175,15 @@ protected boolean hasNextWithoutConstraint() throws IOException {
             groupByPlan.setSlidingStep(slidingStep);
             groupByPlan.setUnit(unit);
             groupByPlan.setDeduplicatedPaths(executePaths);
-            groupByPlan.setDeduplicatedDataTypes(dataTypes);
+            groupByPlan.setDeduplicatedDataTypes(tsDataTypes);
 
 Review comment:
   I think DeviceIterateDataSet needs to do the job of removing duplicate paths

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365675605
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/executor/NewAggregateEngineExecutor.java
 ##########
 @@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.query.executor;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.path.PathException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.MManager;
+import org.apache.iotdb.db.qp.physical.crud.AggregationPlan;
+import org.apache.iotdb.db.query.aggregation.AggreResultData;
+import org.apache.iotdb.db.query.aggregation.AggregateFunction;
+import org.apache.iotdb.db.query.aggregation.impl.LastValueAggrFunc;
+import org.apache.iotdb.db.query.aggregation.impl.MaxTimeAggrFunc;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.dataset.AggreResultDataPointReader;
+import org.apache.iotdb.db.query.dataset.OldEngineDataSetWithoutValueFilter;
+import org.apache.iotdb.db.query.factory.AggreFuncFactory;
+import org.apache.iotdb.db.query.reader.IPointReader;
+import org.apache.iotdb.db.query.reader.IReaderByTimestamp;
+import org.apache.iotdb.db.query.reader.seriesRelated.SeriesDataReaderWithoutValueFilter;
+import org.apache.iotdb.db.query.reader.seriesRelated.SeriesReaderByTimestamp;
+import org.apache.iotdb.db.query.timegenerator.EngineTimeGenerator;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.statistics.Statistics;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.expression.IExpression;
+import org.apache.iotdb.tsfile.read.expression.impl.GlobalTimeExpression;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+public class NewAggregateEngineExecutor {
+
+  private List<Path> selectedSeries;
+  private List<String> aggres;
+  private IExpression expression;
+
+  /**
+   * aggregation batch calculation size.
+   **/
+  private int aggregateFetchSize;
+
+  /**
+   * constructor.
+   */
+  public NewAggregateEngineExecutor(AggregationPlan aggregationPlan) {
+    this.selectedSeries = aggregationPlan.getDeduplicatedPaths();
+    this.aggres = aggregationPlan.getDeduplicatedAggregations();
+    this.expression = aggregationPlan.getExpression();
+    this.aggregateFetchSize = IoTDBDescriptor.getInstance().getConfig().getBatchSize();
+  }
+
+  /**
+   * execute aggregate function with only time filter or no filter.
+   *
+   * @param context query context
+   */
+  public QueryDataSet executeWithoutValueFilter(QueryContext context)
+      throws StorageEngineException, IOException, QueryProcessException {
+    Filter timeFilter = null;
+    if (expression != null) {
+      timeFilter = ((GlobalTimeExpression) expression).getFilter();
+    }
+
+    List<SeriesDataReaderWithoutValueFilter> readersOfSequenceData = new ArrayList<>();
+    List<AggregateFunction> aggregateFunctions = new ArrayList<>();
+    for (int i = 0; i < selectedSeries.size(); i++) {
+      // construct AggregateFunction
+      TSDataType tsDataType = MManager.getInstance()
+          .getSeriesType(selectedSeries.get(i).getFullPath());
+      AggregateFunction function = AggreFuncFactory.getAggrFuncByName(aggres.get(i), tsDataType);
+      function.init();
+      aggregateFunctions.add(function);
+
+      // sequence reader for sealed tsfile, unsealed tsfile, memory
+      SeriesDataReaderWithoutValueFilter newSeriesReaderWithoutValueFilter = new SeriesDataReaderWithoutValueFilter(
+          selectedSeries.get(i), tsDataType, timeFilter, context);
+      readersOfSequenceData.add(newSeriesReaderWithoutValueFilter);
+    }
+
+    List<AggreResultData> aggreResultDataList = new ArrayList<>();
+    //TODO use multi-thread
+    for (int i = 0; i < selectedSeries.size(); i++) {
+      AggreResultData aggreResultData = aggregateWithoutValueFilter(aggregateFunctions.get(i),
+          readersOfSequenceData.get(i), timeFilter);
+      aggreResultDataList.add(aggreResultData);
+    }
+    return constructDataSet(aggreResultDataList);
+  }
+
+  /**
+   * calculation aggregate result with only time filter or no filter for one series.
+   *
+   * @return one series aggregate result data
+   */
+  private AggreResultData aggregateWithoutValueFilter(AggregateFunction function,
+      SeriesDataReaderWithoutValueFilter newSeriesReader, Filter filter)
+      throws IOException, QueryProcessException {
+    if (function instanceof MaxTimeAggrFunc || function instanceof LastValueAggrFunc) {
+      return handleLastMaxTimeWithOutTimeGenerator(function, newSeriesReader, filter);
+    }
+
+    while (newSeriesReader.hasNextChunk()) {
+      if (newSeriesReader.canUseChunkStatistics()) {
+        Statistics chunkStatistics = newSeriesReader.currentChunkStatistics();
+        function.calculateValueFromStatistics(chunkStatistics);
+        if (function.isCalculatedAggregationResult()) {
+          return function.getResult();
+        }
+        newSeriesReader.skipChunkData();
+        continue;
+      }
+      while (newSeriesReader.hasNextPage()) {
+        //cal by pageheader
+        if (newSeriesReader.canUsePageStatistics()) {
+          Statistics pageStatistic = newSeriesReader.currentChunkStatistics();
+          function.calculateValueFromStatistics(pageStatistic);
+          if (function.isCalculatedAggregationResult()) {
+            return function.getResult();
+          }
+          newSeriesReader.skipPageData();
+          continue;
+        }
+        //cal by pagedata
+        while (newSeriesReader.hasNextBatch()) {
+          function.calculateValueFromPageData(newSeriesReader.nextBatch());
+          if (function.isCalculatedAggregationResult()) {
+            return function.getResult();
+          }
+        }
+      }
+    }
+    return function.getResult();
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365789183
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > unSeqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+    while (!seqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = seqFileResource.get(0).getStartTimeMap();
+      Long seqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > seqStartTime) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * Because seq data and unseq data intersect, the minimum startTime taken from two files at a time
+   * is used as the reference time to start reading data
+   */
+  private void searchMinimumChunk() {
+    hasCachedNextChunk = true;
+    if (!seqChunkMetadatas.isEmpty() && unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = seqChunkMetadatas.remove(0);
+    } else if (seqChunkMetadatas.isEmpty() && !unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = unseqChunkMetadatas.pollFirst();
+    } else if (!seqChunkMetadatas.isEmpty()) {
+      // seq 和 unseq 的 chunk metadata 都不为空
+      if (seqChunkMetadatas.get(0).getStartTime() <= unseqChunkMetadatas.first().getStartTime()) {
+        chunkMetaData = seqChunkMetadatas.remove(0);
+      } else {
+        chunkMetaData = unseqChunkMetadatas.pollFirst();
+      }
+    } else {
+      hasCachedNextChunk = false;
+    }
+  }
+
+  /**
+   * Before reading the chunks, should first clean up all the useless chunks, because in the file
+   * hierarchy, although the files are available, some of the internal chunks are still unavailable
+   *
+   * @throws IOException
+   */
+  private void removeInvalidChunks() throws IOException {
+    //remove seq chunks
+    while (filter != null && (!seqChunkMetadatas.isEmpty() || !seqFileResource.isEmpty())) {
+      while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+      }
+
+      ChunkMetaData metaData = seqChunkMetadatas.get(0);
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        seqChunkMetadatas.remove(0);
+        continue;
+      }
+      break;
+    }
+    while (filter != null && (!unseqChunkMetadatas.isEmpty() || !unseqFileResource.isEmpty())) {
+      while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+      }
+      ChunkMetaData metaData = unseqChunkMetadatas.first();
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        unseqChunkMetadatas.pollFirst();
+        continue;
+      }
+      break;
+    }
+    fillMetadataContainer();
+  }
+
+  /**
+   * Because there may be too many files in the scenario used by the user, we cannot open all the
+   * chunks at once, which may OOM, so we can only fill one file at a time when needed. This
+   * approach is likely to be ubiquitous, but it keeps the system running smoothly
+   *
+   * @throws IOException
+   */
+  private void fillMetadataContainer() throws IOException {
+    while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+      seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+    }
+    while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+      unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+    }
+  }
+
+  /**
+   * Before calling this method, you should make sure that all the intersecting files are filled in
+   * the container, because the files intersect, but some chunks may still be useless, so you need
+   * to clean up all the unused chunks and populate the container. It should be noted that this
+   * chunk collection is not in order, and all chunks should be used at once
+   */
+  private void fillOverlappedChunks() throws IOException {
+    while (!unseqChunkMetadatas.isEmpty()) {
+      long startTime = unseqChunkMetadatas.first().getStartTime();
+
+      if (chunkMetaData.getEndTime() > startTime) {
+        ChunkMetaData metaData = unseqChunkMetadatas.pollFirst();
+        IChunkReader chunkReader = initChunkReader(metaData);
+        //When data points overlap, there should be a weight
+        overlappedChunkReader.add(new VersionPair<>(metaData.getVersion(), chunkReader));
+        continue;
+      }
+      break;
+    }
+    while (!seqChunkMetadatas.isEmpty()) {
+      long startTime = seqChunkMetadatas.get(0).getStartTime();
+
+      if (chunkMetaData.getEndTime() > startTime) {
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365669903
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/dataset/NewEngineDataSetWithoutValueFilter.java
 ##########
 @@ -147,7 +152,7 @@ public NewEngineDataSetWithoutValueFilter(List<Path> paths, List<TSDataType> dat
   private void init() throws InterruptedException {
     timeHeap = new TreeSet<>();
     for (int i = 0; i < seriesReaderWithoutValueFilterList.size(); i++) {
-      ManagedSeriesReader reader = seriesReaderWithoutValueFilterList.get(i);
+      RawDataReaderWithoutValueFilter reader = seriesReaderWithoutValueFilterList.get(i);
 
 Review comment:
   done, but only batch was kept
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365578882
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.DiskChunkReader;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      filter = queryDataSource.setTTL(filter);
+      this.filter = filter;
+    }
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    if (filter != null) {
+      this.filter = filter;
+    }
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles(seriesPath, filter);
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  protected boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new DiskChunkReader(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader.addReaderWithPriority(new DiskChunkReader(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  protected BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    return null;
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   *
+   * @param seriesPath
+   * @param filter
+   */
+  private void removeInvalidFiles(Path seriesPath, Filter filter) {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      TsFileResource tsFileResource = seqFileResource.get(0);
+      long startTime = tsFileResource.getStartTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      long endTime = tsFileResource.getEndTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+      if (!filter.satisfyStartEndTime(startTime, endTime)) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      TsFileResource tsFileResource = unseqFileResource.first();
+      Long startTime = tsFileResource.getStartTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      Long endTime = tsFileResource.getEndTimeMap()
+          .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+      if (!filter.satisfyStartEndTime(startTime, endTime)) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
+    while (!unseqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = unseqFileResource.first().getStartTimeMap();
+      Long unSeqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > unSeqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+    while (!seqFileResource.isEmpty()) {
+      Map<String, Long> startTimeMap = seqFileResource.get(0).getStartTimeMap();
+      Long seqStartTime = startTimeMap.getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+      if (chunkMetaData.getEndTime() > seqStartTime) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * Because seq data and unseq data intersect, the minimum startTime taken from two files at a time
+   * is used as the reference time to start reading data
+   */
+  private void searchMinimumChunk() {
+    hasCachedNextChunk = true;
+    if (!seqChunkMetadatas.isEmpty() && unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = seqChunkMetadatas.remove(0);
+    } else if (seqChunkMetadatas.isEmpty() && !unseqChunkMetadatas.isEmpty()) {
+      chunkMetaData = unseqChunkMetadatas.pollFirst();
+    } else if (!seqChunkMetadatas.isEmpty()) {
+      // seq 和 unseq 的 chunk metadata 都不为空
+      if (seqChunkMetadatas.get(0).getStartTime() <= unseqChunkMetadatas.first().getStartTime()) {
+        chunkMetaData = seqChunkMetadatas.remove(0);
+      } else {
+        chunkMetaData = unseqChunkMetadatas.pollFirst();
+      }
+    } else {
+      hasCachedNextChunk = false;
+    }
+  }
+
+  /**
+   * Before reading the chunks, should first clean up all the useless chunks, because in the file
+   * hierarchy, although the files are available, some of the internal chunks are still unavailable
+   *
+   * @throws IOException
+   */
+  private void removeInvalidChunks() throws IOException {
+    //remove seq chunks
+    while (filter != null && (!seqChunkMetadatas.isEmpty() || !seqFileResource.isEmpty())) {
+      while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+        seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+      }
+
+      ChunkMetaData metaData = seqChunkMetadatas.get(0);
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        seqChunkMetadatas.remove(0);
+        continue;
+      }
+      break;
+    }
+    while (filter != null && (!unseqChunkMetadatas.isEmpty() || !unseqFileResource.isEmpty())) {
+      while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+        unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+      }
+      ChunkMetaData metaData = unseqChunkMetadatas.first();
+      if (!filter.satisfyStartEndTime(metaData.getStartTime(), metaData.getEndTime())) {
+        unseqChunkMetadatas.pollFirst();
+        continue;
+      }
+      break;
+    }
+    fillMetadataContainer();
+  }
+
+  /**
+   * Because there may be too many files in the scenario used by the user, we cannot open all the
+   * chunks at once, which may OOM, so we can only fill one file at a time when needed. This
+   * approach is likely to be ubiquitous, but it keeps the system running smoothly
+   *
+   * @throws IOException
+   */
+  private void fillMetadataContainer() throws IOException {
+    while (seqChunkMetadatas.isEmpty() && !seqFileResource.isEmpty()) {
+      seqChunkMetadatas.addAll(loadChunkMetadatas(seqFileResource.remove(0)));
+    }
+    while (unseqChunkMetadatas.isEmpty() && !unseqFileResource.isEmpty()) {
+      unseqChunkMetadatas.addAll(loadChunkMetadatas(unseqFileResource.pollFirst()));
+    }
+  }
+
+  /**
+   * Before calling this method, you should make sure that all the intersecting files are filled in
+   * the container, because the files intersect, but some chunks may still be useless, so you need
+   * to clean up all the unused chunks and populate the container. It should be noted that this
+   * chunk collection is not in order, and all chunks should be used at once
+   */
+  private void fillOverlappedChunks() throws IOException {
+    while (!unseqChunkMetadatas.isEmpty()) {
+      long startTime = unseqChunkMetadatas.first().getStartTime();
+
+      if (chunkMetaData.getEndTime() > startTime) {
+        ChunkMetaData metaData = unseqChunkMetadatas.pollFirst();
+        IChunkReader chunkReader = initChunkReader(metaData);
+        //When data points overlap, there should be a weight
+        overlappedChunkReader.add(new VersionPair<>(metaData.getVersion(), chunkReader));
+        continue;
+      }
+      break;
+    }
+    while (!seqChunkMetadatas.isEmpty()) {
+      long startTime = seqChunkMetadatas.get(0).getStartTime();
+
+      if (chunkMetaData.getEndTime() > startTime) {
+        ChunkMetaData metaData = seqChunkMetadatas.remove(0);
+        IChunkReader chunkReader = initChunkReader(metaData);
+        overlappedChunkReader.add(new VersionPair<>(metaData.getVersion(), chunkReader));
+        continue;
+      }
+      break;
+    }
+  }
+
+  /**
+   * This is just a fake page container, because no matter how chaotic the situation is, the data in
+   * a chunk is always in order, so when the first page of the chunk cannot be used, there is no
+   * need to look at the whole chunk.
+   */
+  private void fillOverlappedPages() {
+    currentPage = chunkReader.nextPageHeader();
+    hasCachedNextPage = true;
+    latestDirectlyOverlappedPageEndTime = currentPage.getEndTime();
+    while (!overlappedChunkReader.isEmpty()) {
+      VersionPair<IChunkReader> iChunkReader = overlappedChunkReader.get(0);
+      if (currentPage.getEndTime() > iChunkReader.data.nextPageHeader().getStartTime()) {
+        overlappedPages.add(overlappedChunkReader.remove(0));
+      }
+    }
+  }
+
+  private class VersionPair<T> {
 
 Review comment:
   Actually I'm a little curious whether this structure can be omitted. For example, order the chunkReaders by the version of metaData, and then use an int value  which increases from 1 as the `priority` value (You can refer to  class `NewUnseqResourceMergeReader`). In this way, we can not only avoid use `VersionPair`, but also use **int** type of `priority` instead of **long**.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365669163
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/dataset/DeviceIterateDataSet.java
 ##########
 @@ -175,15 +175,15 @@ protected boolean hasNextWithoutConstraint() throws IOException {
             groupByPlan.setSlidingStep(slidingStep);
             groupByPlan.setUnit(unit);
             groupByPlan.setDeduplicatedPaths(executePaths);
-            groupByPlan.setDeduplicatedDataTypes(dataTypes);
+            groupByPlan.setDeduplicatedDataTypes(tsDataTypes);
 
 Review comment:
   Yeah, looks like a bug from before

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365789204
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/AbstractDataReader.java
 ##########
 @@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
+import org.apache.iotdb.db.engine.modification.Modification;
+import org.apache.iotdb.db.engine.querycontext.QueryDataSource;
+import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.ManagedSeriesReader;
+import org.apache.iotdb.db.query.reader.MemChunkLoader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkDataIterator;
+import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.QueryUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.tsfile.file.header.PageHeader;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.controller.ChunkLoaderImpl;
+import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
+import org.apache.iotdb.tsfile.read.filter.basic.Filter;
+import org.apache.iotdb.tsfile.read.reader.IChunkReader;
+import org.apache.iotdb.tsfile.read.reader.chunk.ChunkReader;
+
+/*
+ * This class implements a pause read method, pseudocode is:
+ *
+ *
+ * while(hasNextChunk()){
+ *    while(hasNextPage()){
+ *      while(hasNextBatch()){
+ *        nextBatch()
+ *      }
+ *    }
+ * }
+ */
+public abstract class AbstractDataReader implements ManagedSeriesReader {
+
+  private final QueryDataSource queryDataSource;
+  private final QueryContext context;
+  private final Path seriesPath;
+  private final TSDataType dataType;
+  protected Filter filter;
+
+  private final List<TsFileResource> seqFileResource;
+  private final TreeSet<TsFileResource> unseqFileResource;
+
+  private final List<ChunkMetaData> seqChunkMetadatas = new ArrayList<>();
+  private final TreeSet<ChunkMetaData> unseqChunkMetadatas = new TreeSet<>(
+      Comparator.comparingLong(ChunkMetaData::getStartTime));
+
+  private final List<IChunkLoader> openedChunkLoaders = new LinkedList<>();
+
+  protected boolean hasCachedNextChunk;
+  private boolean isCurrentChunkReaderInit;
+  protected IChunkReader chunkReader;
+  protected ChunkMetaData chunkMetaData;
+
+  protected List<VersionPair<IChunkReader>> overlappedChunkReader = new ArrayList<>();
+  protected List<VersionPair<IChunkReader>> overlappedPages = new ArrayList<>();
+
+  protected boolean hasCachedNextPage;
+  protected PageHeader currentPage;
+
+  private boolean hasCachedNextBatch;
+  protected PriorityMergeReader priorityMergeReader = new PriorityMergeReader();
+  private long latestDirectlyOverlappedPageEndTime = Long.MAX_VALUE;
+
+  private boolean hasRemaining;
+  private boolean managedByQueryManager;
+
+
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context) throws StorageEngineException, IOException {
+    queryDataSource = QueryResourceManager.getInstance()
+        .getQueryDataSource(seriesPath, context);
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, QueryDataSource dataSource)
+      throws IOException {
+    queryDataSource = dataSource;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = queryDataSource.setTTL(filter);
+
+    seqFileResource = queryDataSource.getSeqResources();
+    unseqFileResource = sortUnSeqFileResources(queryDataSource.getUnseqResources());
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+  //for test
+  public AbstractDataReader(Path seriesPath, TSDataType dataType,
+      Filter filter, QueryContext context, List<TsFileResource> resources) throws IOException {
+    this.queryDataSource = null;
+    this.seriesPath = seriesPath;
+    this.context = context;
+    this.dataType = dataType;
+
+    this.filter = filter;
+
+    this.seqFileResource = resources;
+    this.unseqFileResource = new TreeSet<>();
+
+    removeInvalidFiles();
+    fillMetadataContainer();
+  }
+
+
+  protected boolean hasNextChunk() throws IOException {
+    if (hasCachedNextChunk) {
+      return true;
+    }
+    removeInvalidChunks();
+    searchMinimumChunk();
+    //When the new chunk cannot be found, it is time to end all methods
+    if (!hasCachedNextChunk) {
+      return false;
+    }
+    fillOverlappedFiles();
+    fillOverlappedChunks();
+    return hasCachedNextChunk;
+  }
+
+
+  protected boolean hasNextPage() throws IOException {
+    if (hasCachedNextPage) {
+      return true;
+    }
+    if (!isCurrentChunkReaderInit) {
+      chunkReader = initChunkReader(chunkMetaData);
+      isCurrentChunkReaderInit = true;
+    }
+
+    if (isCurrentChunkReaderInit && chunkReader.hasNextSatisfiedPage()) {
+      fillOverlappedPages();
+      return hasCachedNextPage;
+    }
+
+    isCurrentChunkReaderInit = false;
+    chunkMetaData.getChunkLoader().close();
+    hasCachedNextChunk = hasCachedNextPage;
+    return hasCachedNextPage;
+  }
+
+
+  public boolean hasNextBatch() throws IOException {
+    if (hasCachedNextBatch) {
+      return true;
+    }
+    if (chunkReader.hasNextSatisfiedPage()) {
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(chunkReader), chunkMetaData.getVersion());
+      hasCachedNextBatch = true;
+    }
+    for (int i = 0; i < overlappedPages.size(); i++) {
+      VersionPair<IChunkReader> reader = overlappedPages.get(i);
+      priorityMergeReader
+          .addReaderWithPriority(new ChunkDataIterator(reader.data), reader.version);
+      hasCachedNextBatch = true;
+    }
+    overlappedPages.clear();
+
+    hasCachedNextPage = hasCachedNextBatch;
+    return hasCachedNextBatch;
+  }
+
+  public BatchData nextBatch() throws IOException {
+    if (priorityMergeReader.hasNext()) {
+      hasCachedNextBatch = false;
+      return nextOverlappedPage();
+    }
+    throw new IOException("no next data");
+  }
+
+
+  protected BatchData nextOverlappedPage() throws IOException {
+    BatchData batchData = new BatchData(dataType);
+    while (priorityMergeReader.hasNext()) {
+      TimeValuePair timeValuePair = priorityMergeReader.current();
+      //TODO should add a batchSize to limit the number of reads per time
+      if (timeValuePair.getTimestamp() > latestDirectlyOverlappedPageEndTime) {
+        break;
+      }
+      batchData.putAnObject(timeValuePair.getTimestamp(), timeValuePair.getValue().getValue());
+      priorityMergeReader.next();
+    }
+    return batchData;
+  }
+
+  private IChunkReader initChunkReader(ChunkMetaData metaData) throws IOException {
+    if (metaData == null) {
+      return null;
+    }
+    IChunkReader chunkReader = null;
+    IChunkLoader chunkLoader = metaData.getChunkLoader();
+    openedChunkLoaders.add(chunkLoader);
+    if (chunkLoader instanceof MemChunkLoader) {
+      MemChunkLoader memChunkLoader = (MemChunkLoader) chunkLoader;
+      chunkReader = new MemChunkReader(memChunkLoader.getChunk(), filter);
+    } else {
+      Chunk chunk = chunkLoader.getChunk(metaData);
+      chunkReader = new ChunkReader(chunk, filter);
+      chunkReader.hasNextSatisfiedPage();
+    }
+    return chunkReader;
+  }
+
+  private List<ChunkMetaData> loadChunkMetadatas(TsFileResource resource) throws IOException {
+    List<ChunkMetaData> currentChunkMetaDataList;
+    if (resource == null) {
+      return new ArrayList<>();
+    }
+    if (resource.isClosed()) {
+      currentChunkMetaDataList = DeviceMetaDataCache.getInstance().get(resource, seriesPath);
+    } else {
+      currentChunkMetaDataList = resource.getChunkMetaDataList();
+    }
+    List<Modification> pathModifications = context
+        .getPathModifications(resource.getModFile(), seriesPath.getFullPath());
+    for (ChunkMetaData data : currentChunkMetaDataList) {
+      if (data.getChunkLoader() == null) {
+        data.setChunkLoader(
+            new ChunkLoaderImpl(new TsFileSequenceReader(resource.getFile().getAbsolutePath())));
+      }
+    }
+    if (!pathModifications.isEmpty()) {
+      QueryUtils.modifyChunkMetaData(currentChunkMetaDataList, pathModifications);
+    }
+    ReadOnlyMemChunk readOnlyMemChunk = resource.getReadOnlyMemChunk();
+    if (readOnlyMemChunk != null) {
+      currentChunkMetaDataList.add(readOnlyMemChunk.getChunkMetaData());
+    }
+    return currentChunkMetaDataList;
+  }
+
+  private TreeSet<TsFileResource> sortUnSeqFileResources(List<TsFileResource> tsFileResources) {
+    TreeSet<TsFileResource> unseqTsFilesSet = new TreeSet<>((o1, o2) -> {
+      Map<String, Long> startTimeMap = o1.getStartTimeMap();
+      Long minTimeOfO1 = startTimeMap.get(seriesPath.getDevice());
+      Map<String, Long> startTimeMap2 = o2.getStartTimeMap();
+      Long minTimeOfO2 = startTimeMap2.get(seriesPath.getDevice());
+
+      return Long.compare(minTimeOfO1, minTimeOfO2);
+    });
+    unseqTsFilesSet.addAll(tsFileResources);
+    return unseqTsFilesSet;
+  }
+
+  @Override
+  public boolean isManagedByQueryManager() {
+    return managedByQueryManager;
+  }
+
+  @Override
+  public void setManagedByQueryManager(boolean managedByQueryManager) {
+    this.managedByQueryManager = managedByQueryManager;
+  }
+
+  @Override
+  public boolean hasRemaining() {
+    return hasRemaining;
+  }
+
+  @Override
+  public void setHasRemaining(boolean hasRemaining) {
+    this.hasRemaining = hasRemaining;
+  }
+
+  /**
+   * Because you get a list of all the files, some files are not necessary when filters exist. This
+   * method filters out the available data files based on the filter
+   */
+  private void removeInvalidFiles() {
+    //filter seq files
+    while (filter != null && !seqFileResource.isEmpty()) {
+      if (!isValid(seqFileResource.get(0))) {
+        seqFileResource.remove(0);
+        continue;
+      }
+      break;
+    }
+    //filter unseq files
+    while (filter != null && !unseqFileResource.isEmpty()) {
+      if (!isValid(unseqFileResource.first())) {
+        unseqFileResource.pollFirst();
+        continue;
+      }
+      break;
+    }
+  }
+
+  private boolean isValid(TsFileResource tsFileResource) {
+    long startTime = tsFileResource.getStartTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MIN_VALUE);
+    long endTime = tsFileResource.getEndTimeMap()
+        .getOrDefault(seriesPath.getDevice(), Long.MAX_VALUE);
+    return filter.satisfyStartEndTime(startTime, endTime);
+  }
+
+  /**
+   * unseq file is a very special file that intersects not only with an ordered file, but also with
+   * another unseq file. So we need a way to find all the files that might be used to intersect the
+   * current measurement point.
+   */
+  private void fillOverlappedFiles() throws IOException {
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365649095
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/reader/seriesRelated/IRawReader.java
 ##########
 @@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.reader.seriesRelated;
+
+import java.io.IOException;
+import org.apache.iotdb.tsfile.read.common.BatchData;
+
+/**
+ * @Author: LiuDaWei
+ * @Create: 2020年01月06日
+ */
+public interface IRawReader {
 
 Review comment:
   rename `IRawReader` to `IRawDataReader`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

liutaohua commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365675001
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/dataset/groupby/GroupByWithoutValueFilterDataSet.java
 ##########
 @@ -58,7 +54,6 @@ public GroupByWithoutValueFilterDataSet(QueryContext context, GroupByPlan groupB
       throws PathException, IOException, StorageEngineException {
     super(context, groupByPlan);
 
-    this.unSequenceReaderList = new ArrayList<>();
     this.sequenceReaderList = new ArrayList<>();
     this.timeFilter = null;
     this.hasCachedSequenceDataList = new ArrayList<>();
 
 Review comment:
   done
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #692: add some impl of NewSeriesReaderWithoutValueFilter
URL: https://github.com/apache/incubator-iotdb/pull/692#discussion_r365581021
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/query/aggregation/impl/FirstValueAggrFunc.java
 ##########
 @@ -45,6 +46,42 @@ public AggreResultData getResult() {
     return resultData;
   }
 
+  @Override
+  public void calculateValueFromStatistics(Statistics chunkStatistics)
+      throws QueryProcessException {
+    if (resultData.isSetTime()) {
+      return;
+    }
+
+    Object firstVal = chunkStatistics.getFirstValue();
+    if (firstVal == null) {
+      throw new QueryProcessException("ChunkMetaData contains no FIRST value");
+    }
+    resultData.putTimeAndValue(0, firstVal);
+  }
+
+  @Override
+  public void calculateValueFromPageData(BatchData dataInThisPage) throws IOException {
+    if (resultData.isSetTime()) {
+      return;
+    }
+    if (dataInThisPage.hasCurrent()) {
+      resultData.putTimeAndValue(0, dataInThisPage.currentValue());
+    }
+  }
+
+  @Override
+  public void calculateValueFromPageData(BatchData dataInThisPage, long bound) throws IOException {
+    if (resultData.isSetTime()) {
+      return;
+    }
+    if (dataInThisPage.hasCurrent() && dataInThisPage.currentTime() < bound) {
+      resultData.putTimeAndValue(0, dataInThisPage.currentValue());
+      dataInThisPage.next();
+      return;
 
 Review comment:
   `return` is unnecessary as the last statement in a `void` method

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services