[GitHub] [iotdb] Alima777 commented on a diff in pull request #7646: [IOTDB-4666]Get the devices on one leaf node at a time while compacting

[GitBox <gi...@apache.org>]

Alima777 commented on code in PR #7646:
URL: https://github.com/apache/iotdb/pull/7646#discussion_r1009111066


##########
tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileDeviceIterator.java:
##########
@@ -25,19 +25,28 @@
 
 import java.io.IOException;
 import java.util.Iterator;
+import java.util.List;
 import java.util.NoSuchElementException;
 import java.util.Queue;
 
 public class TsFileDeviceIterator implements Iterator<Pair<String, Boolean>> {
   private final TsFileSequenceReader reader;
+
+  // device -> firstMeasurmentNode offset
   private final Queue<Pair<String, Pair<Long, Long>>> queue;
   private Pair<String, Boolean> currentDevice = null;
   private MetadataIndexNode measurementNode;
 
+  // <startOffset, endOffset>, device leaf node offset in this file
+  private final List<Pair<Long, Long>> leafDeviceNodeOffsetList;

Review Comment:
   Try to replace Pair<Long, Long> with 'long[2]'



##########
tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java:
##########
@@ -633,37 +633,108 @@ private List<String> getAllDevices(MetadataIndexNode metadataIndexNode) throws I
 
   /**
    * @return an iterator of "device, isAligned" list, in which names of devices are ordered in
-   *     dictionary order, and isAligned represents whether the device is aligned
+   *     dictionary order, and isAligned represents whether the device is aligned. Only read devices
+   *     on one device leaf node each time to save memory.
    */
   public TsFileDeviceIterator getAllDevicesIteratorWithIsAligned() throws IOException {
     readFileMetadata();
-
-    MetadataIndexNode metadataIndexNode = tsFileMetaData.getMetadataIndex();
     Queue<Pair<String, Pair<Long, Long>>> queue = new LinkedList<>();
-    getAllDevicesWithIsAligned(metadataIndexNode, queue);
+    List<Pair<Long, Long>> leafDeviceNodeOffsets = new ArrayList<>();
+    MetadataIndexNode metadataIndexNode = tsFileMetaData.getMetadataIndex();
+    if (metadataIndexNode.getNodeType().equals(MetadataIndexNodeType.LEAF_DEVICE)) {
+      // the first node of index tree is device leaf node, then get the devices directly
+      getDevicesOfLeafNode(metadataIndexNode, queue);
+    } else {
+      // get all device leaf node offset
+      getAllDeviceLeafNodeOffset(metadataIndexNode, leafDeviceNodeOffsets);
+    }
 
-    return new TsFileDeviceIterator(this, queue);
+    return new TsFileDeviceIterator(this, leafDeviceNodeOffsets, queue);
   }
 
-  private void getAllDevicesWithIsAligned(
-      MetadataIndexNode metadataIndexNode, Queue<Pair<String, Pair<Long, Long>>> queue)
+  /**
+   * Get devices and first measurement node offset.
+   *
+   * @param startOffset start offset of device leaf node
+   * @param endOffset end offset of device leaf node
+   * @param measurementNodeOffsetQueue device -> first measurement node offset
+   */
+  public void getDevicesAndEntriesOfOneLeafNode(
+      Long startOffset,
+      Long endOffset,
+      Queue<Pair<String, Pair<Long, Long>>> measurementNodeOffsetQueue)

Review Comment:
   Use primitive type long instead of `Long` as far as possible



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org