You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "shuwenwei (via GitHub)" <gi...@apache.org> on 2023/07/26 04:18:38 UTC
[GitHub] [iotdb] shuwenwei opened a new pull request, #10691: [To rel/1.2] add memory estimator on inner space compaction
shuwenwei opened a new pull request, #10691:
URL: https://github.com/apache/iotdb/pull/10691
## Description
For in-space merge, the memory evaluation process is added. At the beginning of task execution, the system evaluates the memory size of the task to determine whether the evaluation result is smaller than the configured maximum memory size. If the evaluation result is smaller than the configured maximum memory size, the task can be executed normally. Added two memory estimators, the ReadChunkEstimator and the FastCompactionEstimator, which will be used when performer of the merge task is ReadChunk. Otherwise, use FastCompactionEstimator
see https://github.com/apache/iotdb/pull/10688
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #10691: [To rel/1.2] add memory estimator on inner space compaction
Posted by "xingtanzjr (via GitHub)" <gi...@apache.org>.
xingtanzjr commented on code in PR #10691:
URL: https://github.com/apache/iotdb/pull/10691#discussion_r1275893106
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/task/InnerSpaceCompactionTask.java:
##########
@@ -456,9 +476,24 @@ public boolean checkValidAndSetMerging() {
return false;
}
}
+ if (innerSpaceEstimator != null) {
+ memoryCost = innerSpaceEstimator.estimateInnerCompactionMemory(selectedTsFileResourceList);
+ }
+ SystemInfo.getInstance().addCompactionMemoryCost(memoryCost, 60);
+ SystemInfo.getInstance().addCompactionFileNum(selectedTsFileResourceList.size(), 60);
} catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ LOGGER.warn("Interrupted when allocating memory for compaction", e);
+ Thread.currentThread().interrupt();
+ } else if (e instanceof CompactionMemoryNotEnoughException) {
+ LOGGER.info("No enough memory for current compaction task {}", this, e);
+ } else if (e instanceof CompactionFileCountExceededException) {
+ LOGGER.info("No enough file num for current compaction task {}", this, e);
+ SystemInfo.getInstance().resetCompactionMemoryCost(memoryCost);
+ }
+ resetCompactionCandidateStatusForAllSourceFiles();
Review Comment:
This line is unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr merged pull request #10691: [To rel/1.2] add memory estimator on inner space compaction
Posted by "xingtanzjr (via GitHub)" <gi...@apache.org>.
xingtanzjr merged PR #10691:
URL: https://github.com/apache/iotdb/pull/10691
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org