You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2022/10/18 10:41:28 UTC
[GitHub] [iotdb] choubenson opened a new pull request, #7646: [IOTDB-4666]Get the devices on one leaf node at a time while compacting
choubenson opened a new pull request, #7646:
URL: https://github.com/apache/iotdb/pull/7646
At present, the system obtains all devices in the source file at one time when compacting, which may cause OOM problems in the scenario of a large number of devices.
To solve this problem, we should read the devices on one device leaf node at a time instead of reading all devices from files at one time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Alima777 commented on a diff in pull request #7646: [IOTDB-4666]Get the devices on one leaf node at a time while compacting
Posted by GitBox <gi...@apache.org>.
Alima777 commented on code in PR #7646:
URL: https://github.com/apache/iotdb/pull/7646#discussion_r1009111066
##########
tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileDeviceIterator.java:
##########
@@ -25,19 +25,28 @@
import java.io.IOException;
import java.util.Iterator;
+import java.util.List;
import java.util.NoSuchElementException;
import java.util.Queue;
public class TsFileDeviceIterator implements Iterator<Pair<String, Boolean>> {
private final TsFileSequenceReader reader;
+
+ // device -> firstMeasurmentNode offset
private final Queue<Pair<String, Pair<Long, Long>>> queue;
private Pair<String, Boolean> currentDevice = null;
private MetadataIndexNode measurementNode;
+ // <startOffset, endOffset>, device leaf node offset in this file
+ private final List<Pair<Long, Long>> leafDeviceNodeOffsetList;
Review Comment:
Try to replace Pair<Long, Long> with 'long[2]'
##########
tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java:
##########
@@ -633,37 +633,108 @@ private List<String> getAllDevices(MetadataIndexNode metadataIndexNode) throws I
/**
* @return an iterator of "device, isAligned" list, in which names of devices are ordered in
- * dictionary order, and isAligned represents whether the device is aligned
+ * dictionary order, and isAligned represents whether the device is aligned. Only read devices
+ * on one device leaf node each time to save memory.
*/
public TsFileDeviceIterator getAllDevicesIteratorWithIsAligned() throws IOException {
readFileMetadata();
-
- MetadataIndexNode metadataIndexNode = tsFileMetaData.getMetadataIndex();
Queue<Pair<String, Pair<Long, Long>>> queue = new LinkedList<>();
- getAllDevicesWithIsAligned(metadataIndexNode, queue);
+ List<Pair<Long, Long>> leafDeviceNodeOffsets = new ArrayList<>();
+ MetadataIndexNode metadataIndexNode = tsFileMetaData.getMetadataIndex();
+ if (metadataIndexNode.getNodeType().equals(MetadataIndexNodeType.LEAF_DEVICE)) {
+ // the first node of index tree is device leaf node, then get the devices directly
+ getDevicesOfLeafNode(metadataIndexNode, queue);
+ } else {
+ // get all device leaf node offset
+ getAllDeviceLeafNodeOffset(metadataIndexNode, leafDeviceNodeOffsets);
+ }
- return new TsFileDeviceIterator(this, queue);
+ return new TsFileDeviceIterator(this, leafDeviceNodeOffsets, queue);
}
- private void getAllDevicesWithIsAligned(
- MetadataIndexNode metadataIndexNode, Queue<Pair<String, Pair<Long, Long>>> queue)
+ /**
+ * Get devices and first measurement node offset.
+ *
+ * @param startOffset start offset of device leaf node
+ * @param endOffset end offset of device leaf node
+ * @param measurementNodeOffsetQueue device -> first measurement node offset
+ */
+ public void getDevicesAndEntriesOfOneLeafNode(
+ Long startOffset,
+ Long endOffset,
+ Queue<Pair<String, Pair<Long, Long>>> measurementNodeOffsetQueue)
Review Comment:
Use primitive type long instead of `Long` as far as possible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] choubenson commented on a diff in pull request #7646: [IOTDB-4666]Get the devices on one leaf node at a time while compacting
Posted by GitBox <gi...@apache.org>.
choubenson commented on code in PR #7646:
URL: https://github.com/apache/iotdb/pull/7646#discussion_r1012452579
##########
tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java:
##########
@@ -633,37 +633,108 @@ private List<String> getAllDevices(MetadataIndexNode metadataIndexNode) throws I
/**
* @return an iterator of "device, isAligned" list, in which names of devices are ordered in
- * dictionary order, and isAligned represents whether the device is aligned
+ * dictionary order, and isAligned represents whether the device is aligned. Only read devices
+ * on one device leaf node each time to save memory.
*/
public TsFileDeviceIterator getAllDevicesIteratorWithIsAligned() throws IOException {
readFileMetadata();
-
- MetadataIndexNode metadataIndexNode = tsFileMetaData.getMetadataIndex();
Queue<Pair<String, Pair<Long, Long>>> queue = new LinkedList<>();
- getAllDevicesWithIsAligned(metadataIndexNode, queue);
+ List<Pair<Long, Long>> leafDeviceNodeOffsets = new ArrayList<>();
+ MetadataIndexNode metadataIndexNode = tsFileMetaData.getMetadataIndex();
+ if (metadataIndexNode.getNodeType().equals(MetadataIndexNodeType.LEAF_DEVICE)) {
+ // the first node of index tree is device leaf node, then get the devices directly
+ getDevicesOfLeafNode(metadataIndexNode, queue);
+ } else {
+ // get all device leaf node offset
+ getAllDeviceLeafNodeOffset(metadataIndexNode, leafDeviceNodeOffsets);
+ }
- return new TsFileDeviceIterator(this, queue);
+ return new TsFileDeviceIterator(this, leafDeviceNodeOffsets, queue);
}
- private void getAllDevicesWithIsAligned(
- MetadataIndexNode metadataIndexNode, Queue<Pair<String, Pair<Long, Long>>> queue)
+ /**
+ * Get devices and first measurement node offset.
+ *
+ * @param startOffset start offset of device leaf node
+ * @param endOffset end offset of device leaf node
+ * @param measurementNodeOffsetQueue device -> first measurement node offset
+ */
+ public void getDevicesAndEntriesOfOneLeafNode(
+ Long startOffset,
+ Long endOffset,
+ Queue<Pair<String, Pair<Long, Long>>> measurementNodeOffsetQueue)
Review Comment:
Resolved. Use long[2] instead of Pair<Long,Long>.
##########
tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileDeviceIterator.java:
##########
@@ -25,19 +25,28 @@
import java.io.IOException;
import java.util.Iterator;
+import java.util.List;
import java.util.NoSuchElementException;
import java.util.Queue;
public class TsFileDeviceIterator implements Iterator<Pair<String, Boolean>> {
private final TsFileSequenceReader reader;
+
+ // device -> firstMeasurmentNode offset
private final Queue<Pair<String, Pair<Long, Long>>> queue;
private Pair<String, Boolean> currentDevice = null;
private MetadataIndexNode measurementNode;
+ // <startOffset, endOffset>, device leaf node offset in this file
+ private final List<Pair<Long, Long>> leafDeviceNodeOffsetList;
Review Comment:
Resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin merged pull request #7646: [IOTDB-4666]Get the devices on one leaf node at a time while compacting
Posted by GitBox <gi...@apache.org>.
qiaojialin merged PR #7646:
URL: https://github.com/apache/iotdb/pull/7646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org